Safety IndexEvaluationsGray Swan IPI Arena kit
Gray Swan IPI Arena kit
Adversarial robustness · Prompt injectionCounts in the indexBuilt, not yet run
Whether an agent using tools, code or a browser carries out a harmful action and hides it when hostile content is planted in what it reads, using the public Gray Swan IPI Arena kit.
- Metric
- % attack success. Lower is better.
- Dataset
- github.com/GraySwanAI/ipi_arena_os · 41 behaviours; 95 public strings / 28 behaviours · MIT
- Grader
- LLM judge + WorldSim tool simulator (Gemini 2.5 Flash default) (the paper used Gray Swan judge kit)
- Status
- Built, not yet run. Measured on 7 models, 0 readings by SAI.
- Replicability
- public
- Notes
- Third-party rows come from the Gray Swan IPI Arena with human attackers; SAI rows will use the public attack kit.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 0.5%n = 11,969 | 100 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Claude Opus 4.5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere | |
| 1.0%n = 13,603 | 99 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Claude Sonnet 4.5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere | |
| 1.3%n = 13,696 | 99 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Claude Haiku 4.5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere | |
| 2.0%n = 13,143 | 98 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'GPT-5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere | |
| 2.5%n = 8,096 | 98 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'GPT-5.1'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere | |
| 2.9%n = 12,526 | 97 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Grok 4'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere | |
| 5.4%n = 15,026 | 95 | Gray Swan AI IPI Arena (with UK AISI / US CAISI) Independent evaluator · 2026-03-16 | Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'DeepSeek V3.1'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere |