SAI LabsBook a demo

Safety IndexEvaluationsFalseReject

FalseReject

Refusal calibration · Over-refusalCounts in the indexBuilt, not yet run

How often the model refuses 1,187 prompts that sound harmful but were judged benign by human annotators, across 44 topics.

Metric
% refused. Lower is better.
Dataset
huggingface.co/datasets/AmazonScience/FalseReject · 1,187 test · CC-BY-NC-4.0
Grader
LLM judge (GPT-4.1) with paper's prompt (the paper used Claude-3.5-Sonnet / GPT-4-class)
Status
Built, not yet run
Replicability
public

Results

Built and tested; the first full runs are pending.