Safety IndexEvaluationsStrongREJECT (direct, violation rate)
StrongREJECT (direct, violation rate)
Refusal calibration · Harmful requestsBenign twin, shown beside its harm rowRun by SAI
How often the same 313 harmful requests produced harmful content, which is 100 minus the safe-response rate.
- Metric
- % harmful. Lower is better.
- Dataset
- github.com/dsbowen/strong_reject · 313 · MIT
- Grader
- StrongREJECT rubric judge (GPT-4.1) (the paper used GPT-4o-mini)
- Status
- Run by SAI. Measured on 6 models, 6 readings by SAI.
- Replicability
- public
- Twin of
- StrongREJECT (direct)
- Notes
- Same run and same completions as its twin; shown beside it and not counted a second time.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 3.9%n = 310 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal) | |
| 0.6%95% CI 0.2% to 2.3%n = 313 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal) | |
| 2.3%n = 313 | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge azure gpt-4.1-mini (StrongREJECT rubric) | |
| 2.3%n = 313 | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge azure gpt-4.1-mini (StrongREJECT rubric) | |
| 1.3%n = 313 | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge azure gpt-4.1-mini (StrongREJECT rubric) | |
| 1.3%95% CI 0.5% to 3.2%n = 313 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks |