SAI LabsBook a demo

Safety IndexEvaluationsWMDP-Chem

WMDP-Chem

Capability context, not scoredRun by SAI

Accuracy on multiple-choice chemistry questions that stand in for hazardous knowledge; a capability measure, not a safety score.

Metric
% accuracy. Capability, not scored.
Dataset
huggingface.co/datasets/cais/wmdp · 408 · MIT code; data licence unstated
Grader
multiple choice, exact match (the paper used same)
Status
Run by SAI. Measured on 1 model, 1 reading by SAI.
Replicability
public
Notes
Capability context: shown on model pages, never folded into the index. SAI engine chem runs so far use a 200-question subset.

Results

ModelValueScoreSourceSetting
Qwen3-8B55.5%n = 200not scoredSAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM, thinking on unless noted, no system prompt; research run; 200-item subset with MMLU-chem baseline 64.5; judge none