SAI LabsBook a demo

Safety IndexEvaluationsXSTest (HELM Safety)

XSTest (HELM Safety)

Refusal calibration · Over-refusalRetired from the mean (saturated)

How well the model handles the 450 XSTest prompts, helping with the 250 safe ones and refusing the 200 unsafe contrasts, as scored by Stanford HELM.

Metric
HELM score 0-100. Higher is better.
Dataset
github.com/paul-rottger/xstest · 450 (250 safe + 200 unsafe) · CC-BY-4.0
Grader
Llama 3.1 405B and GPT-4o ratings averaged (HELM Safety v1.17.0) (the paper used GPT-4 / string match)
Status
–. Measured on 11 models, 0 readings by SAI.
Replicability
public
Notes
Stanford HELM Safety's XSTest score mixes help on safe prompts with refusal on unsafe ones, so it cannot share a column with the plain safe-prompt refusal rate.

Results

ModelValueScoreSourceSetting
Claude Haiku 4.593.2%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Haiku (20251001)'
Claude Sonnet 4.596.4%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Sonnet (20250929)'
DeepSeek R195.3%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
Gemini 3 Pro97.3%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Gemini 3 Pro (Preview)'
GPT-597.1%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'GPT-5 (2025-08-07)'
GPT-5.198.2%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'GPT-5.1 (2025-11-13)'
gpt-oss-120b92.7%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-120b'
gpt-oss-20b87.2%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-20b'
Grok 496.6%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Llama 4 Maverick96.5%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Llama 4 Maverick (17Bx128E) Instruct FP8'
Qwen3-235B-A22B98.8%not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Qwen3 235B A22B FP8 Throughput'