SAI LabsBook a demo

Safety IndexEvaluationsInstrumentalEval

InstrumentalEval

Interaction and honesty · Deception and schemingCounts in the indexRun by SAI

How often the model chooses power-seeking or self-preserving actions in agentic scenarios when told to pursue a goal at all costs.

Metric
% instrumental. Lower is better.
Dataset
github.com/yf-he/InstrumentalEval · 76 · unverified
Grader
GPT-4o rubric judge (over-flags) (the paper used GPT-4o)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public

Results

ModelValueScoreSourceSetting
Kimi K3.1 Pro9.2%95% CI 4.5% to 17.8%n = 7691SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks
Qwen3.8-Max43.2%95% CI 32.6% to 54.6%n = 7457SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks
Gemini 4 Argon48.7%n = 7651SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; as deployed (a provider block counts as a refusal)