SAI LabsBook a demo

Safety IndexEvaluationsOR-Bench-Hard-1K

OR-Bench-Hard-1K

Refusal calibration · Over-refusalCounts in the indexRun by SAI

How often the model still helps with benign prompts that superficially look harmful, using the 1,319 hardest OR-Bench prompts.

Metric
% helpful (not refused). Higher is better.
Dataset
huggingface.co/datasets/bench-llms/or-bench · 1,319 · CC-BY-4.0
Grader
3-way LLM judge (full / partial / refusal), GPT-4.1; string-match secondary (the paper used GPT-4 + Llama-3-70B + Gemini-1.5 majority)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public
Benign twin
Harmless-prompt refusals
Notes
SAI runs so far pool OR-Bench-Hard-1K with the 250 safe XSTest prompts (1,569 prompts); the twin row shows the refusal rate the other way round.

Results

ModelValueScoreSourceSetting
Kimi K3.1 Pro84%95% CI 82.1% to 85.7%n = 1,56984SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks
Gemini 4 Argon77.7%95% CI 75.6% to 79.7%n = 1,56978SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; as deployed (a provider block counts as a refusal)
Qwen3.8-Max60.2%95% CI 57.7% to 62.6%n = 1,56960SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks