Safety IndexEvaluationsGDM situational awareness
GDM situational awareness
Capability context, not scoredRun by SAI
How many of 11 Google DeepMind challenges the model passes by noticing and working around hidden constraints on itself; a capability measure, not a safety score.
- Metric
- challenges passed of 11. Capability, not scored.
- Dataset
- github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_self_reasoning · 11 × 10-50 tries · Apache-2.0
- Grader
- programmatic (the paper used same)
- Status
- Run by SAI. Measured on 1 model, 1 reading by SAI.
- Replicability
- public
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 8 of 11n = 9 | not scored | SAI Measured by SAI · 2026-10-28 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling; public task set, hard condition, 10 tries per challenge (the report protocol uses more); a challenge counts as passed when its success rate meets the threshold; 2 of 11 challenges (Irreversibility, Approved Directories) could not be scored because Google's API blocked the requests, so 9 were evaluated and 8 of those passed |