Safety IndexEvaluationsScale FORTRESS (risk score)
Scale FORTRESS (risk score)
Refusal calibration · Harmful requestsCounts in the indexBuilt, not yet run
How much dangerous help the model gives on 500 expert-written prompts about chemical, biological, radiological, nuclear and explosive threats, political violence and crime, scored against a rubric written for each prompt.
- Metric
- average risk score 0-100. Lower is better.
- Dataset
- huggingface.co/datasets/ScaleAI/fortress_public · 500 adversarial (+500 benign twins) · CC-BY-4.0
- Grader
- LLM rubric judge; index uses one fixed judge (GPT-4.1), 3-judge panel for audit (the paper used o3 + Claude 3.7 Sonnet + Gemini 2.5 Pro majority)
- Status
- Built, not yet run. Measured on 34 models, 0 readings by SAI.
- Replicability
- public
- Notes
- Prompts and model outputs for the CBRN items are never published; only rates.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 5.8%95% CI 4.5% to 7.0% | 94 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'claude-opus-5-5-max' | |
| 8.2%95% CI 6.3% to 10.2% | 92 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-oss-120b' | |
| 9.5%95% CI 8.1% to 10.8% | 91 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'Claude Haiku 5.5' | |
| 9.6%95% CI 7.5% to 11.7% | 90 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'claude-opus-4-5-20251101-thinking' | |
| 10.8%95% CI 9.6% to 12% | 89 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-6-luna' | |
| 12.4% | 88 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'Muse Spark 1.1' (the current hosted version); the original 'Muse Spark' row scored 20.22 | |
| 12.8%95% CI 10.4% to 15.2% | 87 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'claude-sonnet-4-5-20250929-thinking' | |
| 13%95% CI 10.6% to 15.4% | 87 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'claude-opus-4-6-thinking-max' | |
| 13.5%95% CI 12.1% to 14.9% | 87 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-6-sol' | |
| 13.7%95% CI 11.9% to 15.6% | 86 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'Fable-5.1' | |
| 14.8%95% CI 12.6% to 17.1% | 85 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-5.4-pro-2026-03-05' | |
| 16.3%95% CI 14.6% to 17.9% | 84 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-5.5-xhigh' | |
| 16.8%95% CI 14.9% to 18.8% | 83 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'claude-opus-5' | |
| 17.0%95% CI 14.6% to 19.5% | 83 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-5-2025-08-07' | |
| 17.5%95% CI 15.1% to 19.9% | 83 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-5.2-2025-12-11' | |
| 17.6%95% CI 14.8% to 20.4% | 82 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-oss-20b' | |
| 18.2%95% CI 16.5% to 19.9% | 82 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-6.1-terra' | |
| 20.1%95% CI 18.3% to 22.0% | 80 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-5.6-sol' | |
| 21.6%95% CI 19.7% to 23.5% | 78 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'qwen3p8-2p4t-a95b xhigh' | |
| 23.1%95% CI 20.9% to 25.2% | 77 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'mistral-large-4' | |
| 25.7%95% CI 22.8% to 28.7% | 74 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gpt-5.1-thinking' | |
| 26.6%95% CI 24.5% to 28.8% | 73 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'kimi-k3 max' | |
| 27.2%95% CI 25.1% to 29.3% | 73 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'deepseek-v4p1-flash-1017' | |
| 28.2%95% CI 25.9% to 30.5% | 72 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'glm-5p3 max' | |
| 29.8%95% CI 26.5% to 33.0% | 70 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gemini-3.1-pro-preview' | |
| 40.1%95% CI 36.7% to 43.5% | 60 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'Llama 4 Maverick' | |
| 41.1%95% CI 37.5% to 44.7% | 59 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'kimi-k2.5' | |
| 41.7%95% CI 38.2% to 45.2% | 58 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gemini-3-pro-preview' | |
| 51.0%95% CI 47.6% to 54.5% | 49 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'gemini-3.1-flash-lite-preview' | |
| MiniMax M3.1 Flash | 52.1%95% CI 49.7% to 54.5% | 48 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'minimax-m3p1-flash' |
| 59.6%95% CI 56.2% to 63.0% | 40 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'glm-4p5' | |
| 60.5%95% CI 57.2% to 63.9% | 40 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'DeepSeek-V3.1' | |
| 61.4%95% CI 58.0% to 64.8% | 39 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'Qwen3-235B-A22B' | |
| 74.4%95% CI 71.7% to 77.1% | 26 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'DeepSeek R1' |