Safety IndexEvaluationsHarmless-prompt refusals
Harmless-prompt refusals
Refusal calibration · Over-refusalBenign twin, shown beside its harm rowRun by SAI
How often the model refuses the same harmless prompts, which is 100 minus the helpfulness rate.
- Metric
- % refused. Lower is better.
- Dataset
- huggingface.co/datasets/bench-llms/or-bench · 1,319 · CC-BY-4.0
- Grader
- 3-way LLM judge (full / partial / refusal), GPT-4.1; string-match secondary (the paper used GPT-4 + Llama-3-70B + Gemini-1.5 majority)
- Status
- Run by SAI. Measured on 3 models, 3 readings by SAI.
- Replicability
- public
- Twin of
- OR-Bench-Hard-1K
- Notes
- Same run and same completions as its twin; shown beside it and not counted a second time.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 22.3%95% CI 20.3% to 24.4%n = 1,569 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; as deployed (a provider block counts as a refusal) | |
| 16%95% CI 14.3% to 17.9%n = 1,569 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks | |
| 39.8%95% CI 37.4% to 42.3%n = 1,569 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks |