Safety IndexEvaluationsOR-Bench-Hard-1K
OR-Bench-Hard-1K
Refusal calibration · Over-refusalCounts in the indexRun by SAI
How often the model still helps with benign prompts that superficially look harmful, using the 1,319 hardest OR-Bench prompts.
- Metric
- % helpful (not refused). Higher is better.
- Dataset
- huggingface.co/datasets/bench-llms/or-bench · 1,319 · CC-BY-4.0
- Grader
- 3-way LLM judge (full / partial / refusal), GPT-4.1; string-match secondary (the paper used GPT-4 + Llama-3-70B + Gemini-1.5 majority)
- Status
- Run by SAI. Measured on 3 models, 3 readings by SAI.
- Replicability
- public
- Benign twin
- Harmless-prompt refusals
- Notes
- SAI runs so far pool OR-Bench-Hard-1K with the 250 safe XSTest prompts (1,569 prompts); the twin row shows the refusal rate the other way round.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 84%95% CI 82.1% to 85.7%n = 1,569 | 84 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks | |
| 77.7%95% CI 75.6% to 79.7%n = 1,569 | 78 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; as deployed (a provider block counts as a refusal) | |
| 60.2%95% CI 57.7% to 62.6%n = 1,569 | 60 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks |