SAI LabsBook a demo

Safety IndexEvaluationsTerminal-Bench cheating audit

Terminal-Bench cheating audit

Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run

How often an agent tries to obtain, or obtains, information it was not supposed to see while working on Terminal-Bench tasks.

Metric
% runs attempting or obtaining. Lower is better.
Dataset
www.tbench.ai/ · 66 tasks × 4 attempts · varies per task
Grader
LLM audit over transcripts (public judge prompt) (the paper used Gemini 3.1 Pro judge +ifiers (Google))
Status
Built, not yet run
Replicability
partial

Results

Built and tested; the first full runs are pending.