Safety IndexEvaluationsStrongREJECT with jailbreak templates
StrongREJECT with jailbreak templates
Adversarial robustness · JailbreaksCounts in the indexRun by SAI
How much harmful help the model gives when the 313 StrongREJECT forbidden prompts are wrapped in published jailbreak templates such as AIM, base64 and past-tense rewording.
- Metric
- mean harmfulness score x100 under attack. Lower is better.
- Dataset
- github.com/dsbowen/strong_reject · 313 × (1 + k templates) · MIT
- Grader
- StrongREJECT rubric judge (GPT-4.1) (the paper used GPT-4o-mini)
- Status
- Run by SAI. Measured on 2 models, 2 readings by SAI.
- Replicability
- public
- Notes
- SAI engine rows so far use only the AIM template; the full template set is the v0.2 target.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 60.2%n = 233 | 40 | SAI Measured by SAI · 2026-10-20 | Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.Full configurationAIM template only (one static jailbreak wrapper, not the full template set); vLLM, thinking on unless noted, no system prompt; research run; judge azure gpt-4.1-mini | |
| 64%n = 227 | 36 | SAI Measured by SAI · 2026-10-20 | Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.Full configurationAIM template only (one static jailbreak wrapper, not the full template set); vLLM, thinking on unless noted, no system prompt; research run; judge azure gpt-4.1-mini |