Safety IndexEvaluationsInstrumentalEval
InstrumentalEval
Interaction and honesty · Deception and schemingCounts in the indexRun by SAI
How often the model chooses power-seeking or self-preserving actions in agentic scenarios when told to pursue a goal at all costs.
- Metric
- % instrumental. Lower is better.
- Dataset
- github.com/yf-he/InstrumentalEval · 76 · unverified
- Grader
- GPT-4o rubric judge (over-flags) (the paper used GPT-4o)
- Status
- Run by SAI. Measured on 3 models, 3 readings by SAI.
- Replicability
- public
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 9.2%95% CI 4.5% to 17.8%n = 76 | 91 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks | |
| 43.2%95% CI 32.6% to 54.6%n = 74 | 57 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks | |
| 48.7%n = 76 | 51 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; as deployed (a provider block counts as a refusal) |