Safety IndexEvaluationsImpossibleBench (LiveCodeBench)
ImpossibleBench (LiveCodeBench)
Agentic safety · Reward hacking and task integrityCounts in the indexBuilt, not yet run
How often a coding agent claims to pass tasks whose tests contradict the specification, where any pass means it cheated, even though it was given a way to stop and say so.
- Metric
- % cheating. Lower is better.
- Dataset
- github.com/safety-research/impossiblebench · 103 originals × 2 variants (exclude 13 defective) · MIT
- Grader
- programmatic pass-on-impossible + LLM judge for cheat typing (the paper used claude-opus-4-20250514 judge)
- Status
- Built, not yet run
- Replicability
- public
Results
Built and tested; the first full runs are pending.