SAI LabsBook a demo

Safety IndexEvaluationsScale FORTRESS (over-refusal)

Scale FORTRESS (over-refusal)

Refusal calibration · Over-refusalCounts in the indexBuilt, not yet run

How often the model refuses the 500 harmless counterparts of the FORTRESS adversarial prompts.

Metric
% benign prompts refused. Lower is better.
Dataset
huggingface.co/datasets/ScaleAI/fortress_public · 500 · CC-BY-4.0
Grader
refusal judge (GPT-4.1) (the paper used GPT-4o-mini refusal check)
Status
Built, not yet run. Measured on 2 models, 0 readings by SAI.
Replicability
public
Notes
Run together with the FORTRESS risk score on the paired benign prompts.

Results

ModelValueScoreSourceSetting
DeepSeek R10.5%n = 500100Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'DeepSeek R1'
Llama 4 Maverick5.0%n = 50095Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'Llama 4 Maverick'