SAI LabsBook a demo

Safety IndexEvaluationsScale FORTRESS (risk score)

Scale FORTRESS (risk score)

Refusal calibration · Harmful requestsCounts in the indexBuilt, not yet run

How much dangerous help the model gives on 500 expert-written prompts about chemical, biological, radiological, nuclear and explosive threats, political violence and crime, scored against a rubric written for each prompt.

Metric
average risk score 0-100. Lower is better.
Dataset
huggingface.co/datasets/ScaleAI/fortress_public · 500 adversarial (+500 benign twins) · CC-BY-4.0
Grader
LLM rubric judge; index uses one fixed judge (GPT-4.1), 3-judge panel for audit (the paper used o3 + Claude 3.7 Sonnet + Gemini 2.5 Pro majority)
Status
Built, not yet run. Measured on 34 models, 0 readings by SAI.
Replicability
public
Notes
Prompts and model outputs for the CBRN items are never published; only rates.

Results

ModelValueScoreSourceSetting
Claude Opus 5.55.8%95% CI 4.5% to 7.0%94Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'claude-opus-5-5-max'
gpt-oss-120b8.2%95% CI 6.3% to 10.2%92Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-oss-120b'
Claude Haiku 5.59.5%95% CI 8.1% to 10.8%91Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'Claude Haiku 5.5'
Claude Opus 4.59.6%95% CI 7.5% to 11.7%90Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'claude-opus-4-5-20251101-thinking'
GPT-6 Luna10.8%95% CI 9.6% to 12%89Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-6-luna'
Muse Spark12.4%88Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'Muse Spark 1.1' (the current hosted version); the original 'Muse Spark' row scored 20.22
Claude Sonnet 4.512.8%95% CI 10.4% to 15.2%87Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'claude-sonnet-4-5-20250929-thinking'
Claude Opus 4.613%95% CI 10.6% to 15.4%87Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'claude-opus-4-6-thinking-max'
GPT-6 Sol13.5%95% CI 12.1% to 14.9%87Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-6-sol'
Claude Fable 5.113.7%95% CI 11.9% to 15.6%86Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'Fable-5.1'
GPT-5.4 Pro14.8%95% CI 12.6% to 17.1%85Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-5.4-pro-2026-03-05'
GPT-5.516.3%95% CI 14.6% to 17.9%84Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-5.5-xhigh'
Claude Opus 516.8%95% CI 14.9% to 18.8%83Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'claude-opus-5'
GPT-517.0%95% CI 14.6% to 19.5%83Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-5-2025-08-07'
GPT-5.217.5%95% CI 15.1% to 19.9%83Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-5.2-2025-12-11'
gpt-oss-20b17.6%95% CI 14.8% to 20.4%82Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-oss-20b'
GPT-6.1 Terra18.2%95% CI 16.5% to 19.9%82Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-6.1-terra'
GPT-5.6 Sol20.1%95% CI 18.3% to 22.0%80Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-5.6-sol'
Qwen3.8-2.4T-A95B21.6%95% CI 19.7% to 23.5%78Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'qwen3p8-2p4t-a95b xhigh'
Mistral Large 423.1%95% CI 20.9% to 25.2%77Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'mistral-large-4'
GPT-5.125.7%95% CI 22.8% to 28.7%74Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gpt-5.1-thinking'
Kimi K326.6%95% CI 24.5% to 28.8%73Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'kimi-k3 max'
DeepSeek V4.1 Flash27.2%95% CI 25.1% to 29.3%73Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'deepseek-v4p1-flash-1017'
GLM-5.328.2%95% CI 25.9% to 30.5%72Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'glm-5p3 max'
Gemini 3.1 Pro29.8%95% CI 26.5% to 33.0%70Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gemini-3.1-pro-preview'
Llama 4 Maverick40.1%95% CI 36.7% to 43.5%60Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'Llama 4 Maverick'
Kimi K2.541.1%95% CI 37.5% to 44.7%59Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'kimi-k2.5'
Gemini 3 Pro41.7%95% CI 38.2% to 45.2%58Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gemini-3-pro-preview'
Gemini 3.1 Flash-Lite51.0%95% CI 47.6% to 54.5%49Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'gemini-3.1-flash-lite-preview'
MiniMax M3.1 Flash52.1%95% CI 49.7% to 54.5%48Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'minimax-m3p1-flash'
GLM-4.559.6%95% CI 56.2% to 63.0%40Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'glm-4p5'
DeepSeek V3.160.5%95% CI 57.2% to 63.9%40Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'DeepSeek-V3.1'
Qwen3-235B-A22B61.4%95% CI 58.0% to 64.8%39Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'Qwen3-235B-A22B'
DeepSeek R174.4%95% CI 71.7% to 77.1%26Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'DeepSeek R1'