SAI LabsBook a demo

Safety IndexEvaluationsHarmBench (direct requests)

HarmBench (direct requests)

Refusal calibration · Harmful requestsRetired from the mean (saturated)Built, not yet run

How safely the model answers HarmBench's standard harmful requests when they are asked directly, with no attack.

Metric
safety score 0-100 (HELM). Higher is better.
Dataset
github.com/centerforaisafety/HarmBench · 200 standard (510 total) · MIT
Grader
HarmBench-Llama-2-13b-cls (open) (the paper used same)
Status
Built, not yet run. Measured on 11 models, 0 readings by SAI.
Replicability
public
Notes
Results so far are Stanford HELM Safety scores (0-100, higher is safer), not the attack-success rate the benchmark's own classifier would report.

Results

ModelValueScoreSourceSetting
Claude Haiku 4.595.9%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Claude 4.5 Haiku (20251001)'
Claude Sonnet 4.592%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Claude 4.5 Sonnet (20250929)'
DeepSeek R147.1%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'DeepSeek R1'
Gemini 3 Pro72.5%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Gemini 3 Pro (Preview)'
GPT-597.6%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'GPT-5 (2025-08-07)'
GPT-5.197.6%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'GPT-5.1 (2025-11-13)'
gpt-oss-120b100%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'gpt-oss-120b'
gpt-oss-20b98.7%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'gpt-oss-20b'
Grok 439.7%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Llama 4 Maverick66.1%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Llama 4 Maverick (17Bx128E) Instruct FP8'
Qwen3-235B-A22B56.9%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Qwen3 235B A22B FP8 Throughput'