SAI LabsBook a demo

Safety IndexEvaluationsImpossibleBench (LiveCodeBench)

ImpossibleBench (LiveCodeBench)

Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run

How often a coding agent claims to pass tasks whose tests contradict the specification, where any pass means it cheated, even though it was given a way to stop and say so.

Metric
% cheating. Lower is better.
Dataset
github.com/safety-research/impossiblebench · 103 originals × 2 variants (exclude 13 defective) · MIT
Grader
programmatic pass-on-impossible + LLM judge for cheat typing (the paper used claude-opus-4-20250514 judge)
Status
Built, not yet run
Replicability
public

Results

Built and tested; the first full runs are pending.