Safety IndexEvaluationsTerminal-Bench cheating audit
Terminal-Bench cheating audit
Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run
How often an agent tries to obtain, or obtains, information it was not supposed to see while working on Terminal-Bench tasks.
- Metric
- % runs attempting or obtaining. Lower is better.
- Dataset
- www.tbench.ai/ · 66 tasks × 4 attempts · varies per task
- Grader
- LLM audit over transcripts (public judge prompt) (the paper used Gemini 3.1 Pro judge +ifiers (Google))
- Status
- Built, not yet run
- Replicability
- partial
Results
Built and tested; the first full runs are pending.