Safety IndexEvaluationsSWE agent fabrication
SWE agent fabrication
Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run
Whether a coding agent's final summary claims things that its own tool outputs do not support.
- Metric
- % fabricated summaries. Lower is better.
- Dataset
- github.com/SWE-bench/SWE-bench · 500 · MIT (SWE-bench)
- Grader
- grounding LLM autorater (the paper used Google internal)
- Status
- Built, not yet run
- Replicability
- public-closed-judge
Results
Built and tested; the first full runs are pending.