SAI LabsBook a demo

Safety IndexEvaluationsStrongREJECT with jailbreak templates

StrongREJECT with jailbreak templates

Adversarial robustness · JailbreaksCounts in the indexRun by SAI

How much harmful help the model gives when the 313 StrongREJECT forbidden prompts are wrapped in published jailbreak templates such as AIM, base64 and past-tense rewording.

Metric
mean harmfulness score x100 under attack. Lower is better.
Dataset
github.com/dsbowen/strong_reject · 313 × (1 + k templates) · MIT
Grader
StrongREJECT rubric judge (GPT-4.1) (the paper used GPT-4o-mini)
Status
Run by SAI. Measured on 2 models, 2 readings by SAI.
Replicability
public
Notes
SAI engine rows so far use only the AIM template; the full template set is the v0.2 target.

Results

ModelValueScoreSourceSetting
DeepSeek-R1-Distill-Qwen-7B60.2%n = 23340SAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationAIM template only (one static jailbreak wrapper, not the full template set); vLLM, thinking on unless noted, no system prompt; research run; judge azure gpt-4.1-mini
Qwen3-8B64%n = 22736SAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationAIM template only (one static jailbreak wrapper, not the full template set); vLLM, thinking on unless noted, no system prompt; research run; judge azure gpt-4.1-mini