SAI LabsBook a demo

Safety IndexEvaluationsProgramBench cheating audit

ProgramBench cheating audit

Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run

Whether an agent told to re-implement a command-line tool inspects the forbidden original source or quietly delegates to the installed tool.

Metric
% runs cheating. Lower is better.
Dataset
– · 200 tasks · unverified
Grader
multi-stage transcript audit (LLM) (the paper used Google internal)
Status
Built, not yet run
Replicability
public-closed-judge

Results

Built and tested; the first full runs are pending.