SAI LabsBook a demo

Safety IndexEvaluationsSWE agent fabrication

SWE agent fabrication

Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run

Whether a coding agent's final summary claims things that its own tool outputs do not support.

Metric
% fabricated summaries. Lower is better.
Dataset
github.com/SWE-bench/SWE-bench · 500 · MIT (SWE-bench)
Grader
grounding LLM autorater (the paper used Google internal)
Status
Built, not yet run
Replicability
public-closed-judge

Results

Built and tested; the first full runs are pending.