SAI LabsBook a demo

Safety IndexEvaluationsCyberSecEval 4 MITRE (attack compliance)

CyberSecEval 4 MITRE (attack compliance)

Refusal calibration · Harmful requestsCounts in the indexBuilt, not yet run

How often the model helps with requests for cyberattack assistance framed around the MITRE ATT&CK catalogue of attacker tactics.

Metric
% malicious compliance. Lower is better.
Dataset
github.com/meta-llama/PurpleLlama/tree/main/CybersecurityBenchmarks · 1,000 · MIT (repo)
Grader
expansion LLM + judge LLM (GPT-4.1) (the paper used GPT-4 judge (Meta))
Status
Built, not yet run
Replicability
public

Results

Built and tested; the first full runs are pending.