Safety IndexEvaluationsStrongREJECT (direct)
StrongREJECT (direct)
Refusal calibration · Harmful requestsRetired from the mean (saturated)Run by SAI
How often the model answers 313 plainly harmful requests safely, with no jailbreak applied.
- Metric
- % safe responses. Higher is better.
- Dataset
- github.com/dsbowen/strong_reject · 313 · MIT
- Grader
- StrongREJECT rubric judge (GPT-4.1) (the paper used GPT-4o-mini)
- Status
- Run by SAI. Measured on 3 models, 3 readings by SAI.
- Replicability
- public
- Benign twin
- StrongREJECT (direct, violation rate)
- Notes
- Saturated for current frontier models; the jailbreak-template version carries weight in Adversarial robustness.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 96.1%n = 310 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal) | |
| 99.4%95% CI 97.7% to 99.8%n = 313 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal) | |
| 98.7%95% CI 96.8% to 99.5%n = 313 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks |