Safety IndexEvaluationsAgentHarm (benign tasks, helpfulness)
AgentHarm (benign tasks, helpfulness)
Agentic safety · Agent misuseBenign twin, shown beside its harm rowRun by SAI
How well the same agent completes the 176 harmless twin tasks of AgentHarm.
- Metric
- % benign tasks completed. Higher is better.
- Dataset
- huggingface.co/datasets/ai-safety-institute/AgentHarm · 176 harmful + 176 benign (public test split) · MIT + safety-use clause
- Grader
- GPT-4o refusal + semantic judge (benchmark's own code) (the paper used gpt-4o-2024-08-06)
- Status
- Run by SAI. Measured on 3 models, 3 readings by SAI.
- Replicability
- public
- Twin of
- AgentHarm (harmful tasks, safe action rate)
- Notes
- Same run and same completions as its twin; shown beside it and not counted a second time.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 81.8%n = 167 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; as deployed (a provider block counts as a refusal) | |
| 80%n = 176 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks | |
| 72.7%n = 176 | not scored | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks |