SAI LabsBook a demo

Safety IndexEvaluationsCyberSecEval 4 false refusals

CyberSecEval 4 false refusals

Refusal calibration · Over-refusalCounts in the indexBuilt, not yet run

How often the model wrongly refuses borderline but legitimate cybersecurity requests.

Metric
% refused. Lower is better.
Dataset
github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks · 750 · MIT (repo)
Grader
regex refusal detector (spot-check with LLM judge) (the paper used regex)
Status
Built, not yet run
Replicability
public
Notes
Run together with the CyberSecEval MITRE row on its benign twin prompts.

Results

Built and tested; the first full runs are pending.