Safety IndexEvaluationsHarmBench (direct requests)
HarmBench (direct requests)
Refusal calibration · Harmful requestsRetired from the mean (saturated)Built, not yet run
How safely the model answers HarmBench's standard harmful requests when they are asked directly, with no attack.
- Metric
- safety score 0-100 (HELM). Higher is better.
- Dataset
- github.com/centerforaisafety/HarmBench · 200 standard (510 total) · MIT
- Grader
- HarmBench-Llama-2-13b-cls (open) (the paper used same)
- Status
- Built, not yet run. Measured on 11 models, 0 readings by SAI.
- Replicability
- public
- Notes
- Results so far are Stanford HELM Safety scores (0-100, higher is safer), not the attack-success rate the benchmark's own classifier would report.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 95.9% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Claude 4.5 Haiku (20251001)' | |
| 92% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Claude 4.5 Sonnet (20250929)' | |
| 47.1% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'DeepSeek R1' | |
| 72.5% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Gemini 3 Pro (Preview)' | |
| 97.6% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'GPT-5 (2025-08-07)' | |
| 97.6% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'GPT-5.1 (2025-11-13)' | |
| 100% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'gpt-oss-120b' | |
| 98.7% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'gpt-oss-20b' | |
| 39.7% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Grok 4 (0709)' | |
| 66.1% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Llama 4 Maverick (17Bx128E) Instruct FP8' | |
| 56.9% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Qwen3 235B A22B FP8 Throughput' |