SAI LabsBook a demo

Safety IndexEvaluationsAIR-Bench 2024

AIR-Bench 2024

Refusal calibration · Harmful requestsCounts in the indexRun by SAI

How often the model refuses 5,694 requests drawn from AI regulations and company policies, spanning 314 kinds of risk.

Metric
% refused (safe). Higher is better.
Dataset
huggingface.co/datasets/stanford-crfm/air-bench-2024 · 5,694 · CC-BY-4.0
Grader
category-specific LLM judge prompts (GPT-4.1) (the paper used GPT-4o with AIR-Bench judge prompts)
Status
Run by SAI. Measured on 14 models, 3 readings by SAI.
Replicability
public
Notes
The privacy and election slices of the same run are reported separately under People and society.

Results

ModelValueScoreSourceSetting
Kimi K3.1 Pro93.9%95% CI 91.7% to 95.5%n = 63594SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks
Qwen3.8-Max93.9%95% CI 91.7% to 95.5%n = 63594SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks
Claude Haiku 4.593.2%n = 5,69493Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Claude Haiku 4.5'
Claude Sonnet 4.589.8%n = 5,69490Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Claude Sonnet 4.5'
gpt-oss-120b88%n = 5,69488Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'gpt-oss-120b'
GPT-587.7%n = 5,69488Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'GPT-5'
GPT-5.186.2%n = 5,69486Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'GPT-5.1'
gpt-oss-20b86.0%n = 5,69486Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'gpt-oss-20b'
Gemini 4 Argon77%n = 63677SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; as deployed (a provider block counts as a refusal)
Gemini 3 Pro73.2%n = 5,69473Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Gemini 3 Pro'
Llama 4 Maverick68.6%n = 5,69469Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Llama 4 Maverick'
Qwen3-235B-A22B56.0%n = 5,69456Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Qwen3 235B A22B'
DeepSeek R152.9%n = 5,69453Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'DeepSeek R1'
Grok 444.4%n = 5,69444Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Grok 4'