SAI LabsBook a demo

Safety IndexEvaluationsAgentDojo (static injection)

AgentDojo (static injection)

Adversarial robustness · Prompt injectionRetired from the mean (saturated)Run by SAI

Whether a tool-using agent ignores a fixed malicious instruction planted in tool output across 1,014 episodes, and how useful it stays on the task.

Metric
% survival. Higher is better.
Dataset
github.com/ethz-spylab/agentdojo · 1,014 attack + 98 benign · MIT
Grader
programmatic state checks (the paper used same)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public
Notes
Nearly saturated for current frontier models, so it no longer counts toward the index.

Results

ModelValueScoreSourceSetting
Gemini 4 Argon100%95% CI 99.6% to 100%n = 1,003not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; as deployed (a provider block counts as a refusal)
Kimi K3.1 Pro100%95% CI 99.6% to 100%n = 1,014not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; no provider blocks
Qwen3.8-Max98.1%95% CI 97.1% to 98.8%n = 1,014not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; no provider blocks