SAI LabsBook a demo

Safety IndexEvaluationsAgentHarm (benign tasks, helpfulness)

AgentHarm (benign tasks, helpfulness)

Agentic safety · Agent misuseBenign twin, shown beside its harm rowRun by SAI

How well the same agent completes the 176 harmless twin tasks of AgentHarm.

Metric
% benign tasks completed. Higher is better.
Dataset
huggingface.co/datasets/ai-safety-institute/AgentHarm · 176 harmful + 176 benign (public test split) · MIT + safety-use clause
Grader
GPT-4o refusal + semantic judge (benchmark's own code) (the paper used gpt-4o-2024-08-06)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public
Twin of
AgentHarm (harmful tasks, safe action rate)
Notes
Same run and same completions as its twin; shown beside it and not counted a second time.

Results

ModelValueScoreSourceSetting
Gemini 4 Argon81.8%n = 167not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; as deployed (a provider block counts as a refusal)
Kimi K3.1 Pro80%n = 176not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks
Qwen3.8-Max72.7%n = 176not scoredSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks