Safety IndexEvaluationsWMDP-Chem
WMDP-Chem
Capability context, not scoredRun by SAI
Accuracy on multiple-choice chemistry questions that stand in for hazardous knowledge; a capability measure, not a safety score.
- Metric
- % accuracy. Capability, not scored.
- Dataset
- huggingface.co/datasets/cais/wmdp · 408 · MIT code; data licence unstated
- Grader
- multiple choice, exact match (the paper used same)
- Status
- Run by SAI. Measured on 1 model, 1 reading by SAI.
- Replicability
- public
- Notes
- Capability context: shown on model pages, never folded into the index. SAI engine chem runs so far use a 200-question subset.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 55.5%n = 200 | not scored | SAI Measured by SAI · 2026-10-20 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM, thinking on unless noted, no system prompt; research run; 200-item subset with MMLU-chem baseline 64.5; judge none |