SAI LabsBook a demo

Safety IndexEvaluationsBenign-request over-refusal rate (lab-reported)

Benign-request over-refusal rate (lab-reported)

Refusal calibration · Over-refusalDeveloper-reported

The developer's own measure of how often the model refused harmless prompts in sensitive areas.

Metric
% refused. Lower is better.
Dataset
–
Grader
the developer's own grader (see the model's system card)
Status
–. Measured on 8 models, 0 readings by SAI.
Notes
Lab system cards; internal prompt sets. Developer-reported; never counted.

Results

ModelValueScoreSourceSetting
Claude Fable 5.10.0%developer-reportedAnthropic (Claude Opus 5.5 system card)
Developer-reported · 2026-09-22
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate, API without system prompt; Table 4.1.2.A; claude.ai 0.34%
Claude Haiku 4.50.0%developer-reportedAnthropic (Claude Haiku 4.5 system card)
Developer-reported · 2025-10
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate; Table 2.1.2.A
Claude Opus 4.50.2%developer-reportedAnthropic (Claude Opus 4.5 system card)
Developer-reported · 2025-11
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate, 6 languages; Table 3.1.2.A
Claude Opus 4.60.7%developer-reportedAnthropic (Claude Opus 4.6 system card)
Developer-reported · 2026-02
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate, 7 languages; Table 3.1.2.A
Claude Opus 50.1%developer-reportedAnthropic (Claude Opus 5 system card)
Developer-reported · 2026-07-24
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate, API without system prompt; Table 4.1.2.A; claude.ai 0.47%
Claude Opus 5.50.0%developer-reportedAnthropic (Claude Opus 5.5 system card)
Developer-reported · 2026-09-22
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate, API without system prompt; Table 4.1.2.A; claude.ai 0.38%
Claude Sonnet 4.50.0%developer-reportedAnthropic (Claude Sonnet 4.5 system card)
Developer-reported · 2025-09
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate; Table 2.1.2.A; run without safety classifiers
Claude Sonnet 50.6%developer-reportedAnthropic (Claude Opus 5 system card)
Developer-reported · 2026-07-24
Reported by the model's developer in its own system card or report.
Full configurationoverall over-refusal rate, API without system prompt; Table 4.1.2.A; claude.ai 1.54%