Safety IndexEvaluationsSORRY-Bench
SORRY-Bench
Refusal calibration · Harmful requestsCounts in the indexPlanned
How often the model fulfils 440 unsafe instructions spread evenly across 44 fine-grained categories of harm.
- Metric
- % fulfilled. Lower is better.
- Dataset
- huggingface.co/datasets/sorry-bench/sorry-bench-202503 · 440 base (+8.8k mutations) · custom SORRY-Bench licence (evaluation ok, no redistribution; click-through)
- Grader
- fine-tuned Mistral-7B judge (SORRY-Bench's fine-tuned Mistral-7B judge), local (the paper used same)
- Status
- Planned
- Replicability
- public
- Notes
- Dataset access is being arranged before the first run.
Results
Planned; no measurements yet.