SAI LabsBook a demo

Safety IndexEvaluationsStrongREJECT (direct, violation rate)

StrongREJECT (direct, violation rate)

Refusal calibration · Harmful requestsBenign twin, shown beside its harm rowRun by SAI

How often the same 313 harmful requests produced harmful content, which is 100 minus the safe-response rate.

Metric
% harmful. Lower is better.
Dataset
github.com/dsbowen/strong_reject · 313 · MIT
Grader
StrongREJECT rubric judge (GPT-4.1) (the paper used GPT-4o-mini)
Status
Run by SAI. Measured on 6 models, 6 readings by SAI.
Replicability
public
Twin of
StrongREJECT (direct)
Notes
Same run and same completions as its twin; shown beside it and not counted a second time.

Results

ModelValueScoreSourceSetting
Gemini 4 Argon3.9%n = 310not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal)
Kimi K3.1 Pro0.6%95% CI 0.2% to 2.3%n = 313not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal)
Qwen3.5-4B2.3%n = 313not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge azure gpt-4.1-mini (StrongREJECT rubric)
Qwen3.5-9B2.3%n = 313not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge azure gpt-4.1-mini (StrongREJECT rubric)
Qwen3.8-27B1.3%n = 313not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge azure gpt-4.1-mini (StrongREJECT rubric)
Qwen3.8-Max1.3%95% CI 0.5% to 3.2%n = 313not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks