Safety IndexEvaluationsAIR-Bench 2024
AIR-Bench 2024
Refusal calibration · Harmful requestsCounts in the indexRun by SAI
How often the model refuses 5,694 requests drawn from AI regulations and company policies, spanning 314 kinds of risk.
- Metric
- % refused (safe). Higher is better.
- Dataset
- huggingface.co/datasets/stanford-crfm/air-bench-2024 · 5,694 · CC-BY-4.0
- Grader
- category-specific LLM judge prompts (GPT-4.1) (the paper used GPT-4o with AIR-Bench judge prompts)
- Status
- Run by SAI. Measured on 14 models, 3 readings by SAI.
- Replicability
- public
- Notes
- The privacy and election slices of the same run are reported separately under People and society.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 93.9%95% CI 91.7% to 95.5%n = 635 | 94 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks | |
| 93.9%95% CI 91.7% to 95.5%n = 635 | 94 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks | |
| 93.2%n = 5,694 | 93 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Claude Haiku 4.5' | |
| 89.8%n = 5,694 | 90 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Claude Sonnet 4.5' | |
| 88%n = 5,694 | 88 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'gpt-oss-120b' | |
| 87.7%n = 5,694 | 88 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'GPT-5' | |
| 86.2%n = 5,694 | 86 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'GPT-5.1' | |
| 86.0%n = 5,694 | 86 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'gpt-oss-20b' | |
| 77%n = 636 | 77 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; as deployed (a provider block counts as a refusal) | |
| 73.2%n = 5,694 | 73 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Gemini 3 Pro' | |
| 68.6%n = 5,694 | 69 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Llama 4 Maverick' | |
| 56.0%n = 5,694 | 56 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Qwen3 235B A22B' | |
| 52.9%n = 5,694 | 53 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'DeepSeek R1' | |
| 44.4%n = 5,694 | 44 | Stanford CRFM HELM AIR-Bench v1.19.0 Independent evaluator | Published by Stanford CRFM HELM AIR-Bench v1.19.0.Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Grok 4' |