SAI LabsBook a demo

Safety IndexEvaluationsHarmless-prompt refusals

Harmless-prompt refusals

Refusal calibration · Over-refusalBenign twin, shown beside its harm rowRun by SAI

How often the model refuses the same harmless prompts, which is 100 minus the helpfulness rate.

Metric
% refused. Lower is better.
Dataset
huggingface.co/datasets/bench-llms/or-bench · 1,319 · CC-BY-4.0
Grader
3-way LLM judge (full / partial / refusal), GPT-4.1; string-match secondary (the paper used GPT-4 + Llama-3-70B + Gemini-1.5 majority)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public
Twin of
OR-Bench-Hard-1K
Notes
Same run and same completions as its twin; shown beside it and not counted a second time.

Results

ModelValueScoreSourceSetting
Gemini 4 Argon22.3%95% CI 20.3% to 24.4%n = 1,569not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; as deployed (a provider block counts as a refusal)
Kimi K3.1 Pro16%95% CI 14.3% to 17.9%n = 1,569not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks
Qwen3.8-Max39.8%95% CI 37.4% to 42.3%n = 1,569not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks