SAI LabsBook a demo

Safety IndexEvaluationsSORRY-Bench

SORRY-Bench

Refusal calibration · Harmful requestsCounts in the indexPlanned

How often the model fulfils 440 unsafe instructions spread evenly across 44 fine-grained categories of harm.

Metric
% fulfilled. Lower is better.
Dataset
huggingface.co/datasets/sorry-bench/sorry-bench-202503 · 440 base (+8.8k mutations) · custom SORRY-Bench licence (evaluation ok, no redistribution; click-through)
Grader
fine-tuned Mistral-7B judge (SORRY-Bench's fine-tuned Mistral-7B judge), local (the paper used same)
Status
Planned
Replicability
public
Notes
Dataset access is being arranged before the first run.

Results

Planned; no measurements yet.