Safety IndexEvaluationsProgramBench cheating audit
ProgramBench cheating audit
Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run
Whether an agent told to re-implement a command-line tool inspects the forbidden original source or quietly delegates to the installed tool.
- Metric
- % runs cheating. Lower is better.
- Dataset
- – · 200 tasks · unverified
- Grader
- multi-stage transcript audit (LLM) (the paper used Google internal)
- Status
- Built, not yet run
- Replicability
- public-closed-judge
Results
Built and tested; the first full runs are pending.