Safety IndexEvaluationsFalseReject
FalseReject
Refusal calibration · Over-refusalCounts in the indexBuilt, not yet run
How often the model refuses 1,187 prompts that sound harmful but were judged benign by human annotators, across 44 topics.
- Metric
- % refused. Lower is better.
- Dataset
- huggingface.co/datasets/AmazonScience/FalseReject · 1,187 test · CC-BY-NC-4.0
- Grader
- LLM judge (GPT-4.1) with paper's prompt (the paper used Claude-3.5-Sonnet / GPT-4-class)
- Status
- Built, not yet run
- Replicability
- public
Results
Built and tested; the first full runs are pending.