Safety IndexEvaluationsXSTest (HELM Safety)
XSTest (HELM Safety)
Refusal calibration · Over-refusalRetired from the mean (saturated)
How well the model handles the 450 XSTest prompts, helping with the 250 safe ones and refusing the 200 unsafe contrasts, as scored by Stanford HELM.
- Metric
- HELM score 0-100. Higher is better.
- Dataset
- github.com/paul-rottger/xstest · 450 (250 safe + 200 unsafe) · CC-BY-4.0
- Grader
- Llama 3.1 405B and GPT-4o ratings averaged (HELM Safety v1.17.0) (the paper used GPT-4 / string match)
- Status
- –. Measured on 11 models, 0 readings by SAI.
- Replicability
- public
- Notes
- Stanford HELM Safety's XSTest score mixes help on safe prompts with refusal on unsafe ones, so it cannot share a column with the plain safe-prompt refusal rate.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 93.2% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Haiku (20251001)' | |
| 96.4% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Sonnet (20250929)' | |
| 95.3% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1' | |
| 97.3% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Gemini 3 Pro (Preview)' | |
| 97.1% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'GPT-5 (2025-08-07)' | |
| 98.2% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'GPT-5.1 (2025-11-13)' | |
| 92.7% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-120b' | |
| 87.2% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-20b' | |
| 96.6% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)' | |
| 96.5% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Llama 4 Maverick (17Bx128E) Instruct FP8' | |
| 98.8% | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Qwen3 235B A22B FP8 Throughput' |