SAI LabsBook a demo

Safety IndexEvaluationsStrongREJECT (direct)

StrongREJECT (direct)

Refusal calibration · Harmful requestsRetired from the mean (saturated)Run by SAI

How often the model answers 313 plainly harmful requests safely, with no jailbreak applied.

Metric
% safe responses. Higher is better.
Dataset
github.com/dsbowen/strong_reject · 313 · MIT
Grader
StrongREJECT rubric judge (GPT-4.1) (the paper used GPT-4o-mini)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public
Benign twin
StrongREJECT (direct, violation rate)
Notes
Saturated for current frontier models; the jailbreak-template version carries weight in Adversarial robustness.

Results

ModelValueScoreSourceSetting
Gemini 4 Argon96.1%n = 310not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal)
Kimi K3.1 Pro99.4%95% CI 97.7% to 99.8%n = 313not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal)
Qwen3.8-Max98.7%95% CI 96.8% to 99.5%n = 313not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks