SAI LabsBook a demo

Safety IndexEvaluationsWMDP-Cyber

WMDP-Cyber

Capability context, not scoredRun by SAI

Accuracy on 1,987 multiple-choice questions about offensive cyber knowledge; a capability measure, not a safety score.

Metric
% accuracy. Capability, not scored.
Dataset
huggingface.co/datasets/cais/wmdp · 1,987 · MIT code; data licence unstated
Grader
multiple choice, exact match (the paper used same)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public
Notes
Capability context: shown on model pages, never folded into the index.

Results

ModelValueScoreSourceSetting
Qwen3.5-4B46.6%n = 1,987not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none
Qwen3.5-9B45%n = 1,987not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none
Qwen3.8-27B54.5%n = 1,987not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none