SAI LabsBook a demo

Safety IndexEvaluationsAbstentionBench

AbstentionBench

Interaction and honesty · Factuality and hallucinationCounts in the indexBuilt, not yet run

Whether the model declines to answer questions that are unanswerable, under-specified, based on a false premise or out of date.

Metric
abstention recall. Higher is better.
Dataset
huggingface.co/datasets/facebook/AbstentionBench · 2,104 (fast) / 39,558 · CC-BY-NC-4.0
Grader
LLM abstention detector (the paper used Llama-3.1-8B default)
Status
Built, not yet run
Replicability
public

Results

Built and tested; the first full runs are pending.