SAI LabsBook a demo

Safety IndexEvaluationsGray Swan IPI Arena kit

Gray Swan IPI Arena kit

Adversarial robustness · Prompt injectionCounts in the indexBuilt, not yet run

Whether an agent using tools, code or a browser carries out a harmful action and hides it when hostile content is planted in what it reads, using the public Gray Swan IPI Arena kit.

Metric
% attack success. Lower is better.
Dataset
github.com/GraySwanAI/ipi_arena_os · 41 behaviours; 95 public strings / 28 behaviours · MIT
Grader
LLM judge + WorldSim tool simulator (Gemini 2.5 Flash default) (the paper used Gray Swan judge kit)
Status
Built, not yet run. Measured on 7 models, 0 readings by SAI.
Replicability
public
Notes
Third-party rows come from the Gray Swan IPI Arena with human attackers; SAI rows will use the public attack kit.

Results

ModelValueScoreSourceSetting
Claude Opus 4.50.5%n = 11,969100Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Claude Opus 4.5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
Claude Sonnet 4.51.0%n = 13,60399Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Claude Sonnet 4.5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
Claude Haiku 4.51.3%n = 13,69699Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Claude Haiku 4.5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
GPT-52.0%n = 13,14398Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'GPT-5'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
GPT-5.12.5%n = 8,09698Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'GPT-5.1'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
Grok 42.9%n = 12,52697Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Grok 4'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
DeepSeek V3.15.4%n = 15,02695Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'DeepSeek V3.1'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere