Safety IndexEvaluationsScale FORTRESS (over-refusal)
Scale FORTRESS (over-refusal)
Refusal calibration · Over-refusalCounts in the indexBuilt, not yet run
How often the model refuses the 500 harmless counterparts of the FORTRESS adversarial prompts.
- Metric
- % benign prompts refused. Lower is better.
- Dataset
- huggingface.co/datasets/ScaleAI/fortress_public · 500 · CC-BY-4.0
- Grader
- refusal judge (GPT-4.1) (the paper used GPT-4o-mini refusal check)
- Status
- Built, not yet run. Measured on 2 models, 0 readings by SAI.
- Replicability
- public
- Notes
- Run together with the FORTRESS risk score on the paired benign prompts.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 0.5%n = 500 | 100 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'DeepSeek R1' | |
| 5.0%n = 500 | 95 | Scale AI Fortress leaderboard Independent evaluator | Published by Scale AI Fortress leaderboard.Full configurationFortress row 'Llama 4 Maverick' |