Safety IndexMethodologyv0.2 preview
How the SAI Safety Index is built
The index exists to make one question answerable at a glance, how safely a model behaves, without hiding how the answer was reached. This page gives the structure, the arithmetic, the rules for what counts, and what changed between versions. The same arithmetic runs in the page code, so the charts and this description cannot drift apart.
Structure
Five pillars, equal weights. Each pillar is split into sub-categories that answer one plain question, and each sub-category is measured by one or more public evaluations. The full catalogue, with each evaluation's dataset, grader and status, is on the evaluations page.
| Pillar | Sub-categories | Weight |
|---|---|---|
| Refusal calibration Does the model refuse what it should and help with what it may? | Harmful requests (7 evaluations) Over-refusal (5 evaluations) | 20% |
| Adversarial robustness Do its refusals and instructions hold up when someone tries to get around them? | Jailbreaks (4 evaluations) Prompt injection (8 evaluations) Multilingual safety (5 evaluations) | 20% |
| Agentic safety Does it behave safely when it has tools and autonomy? | Agent misuse (3 evaluations) Reward hacking and task integrity (5 evaluations) Overreach and destructive actions (3 evaluations) | 20% |
| Interaction and honesty Is it honest with the person it is talking to? | Factuality and hallucination (5 evaluations) Honesty under pressure and sycophancy (6 evaluations) Deception and scheming (4 evaluations) Alignment audit (2 evaluations) Manipulation of the user (3 evaluations) | 20% |
| People and society Does it protect vulnerable people and the public interest? | Self-harm and wellbeing (3 evaluations) Child safety (2 evaluations) Bias, fairness and neutrality (3 evaluations) Privacy: memorisation and leakage (2 evaluations) Privacy: surveillance assistance (4 evaluations) Elections and influence operations (4 evaluations) | 20% |
Arithmetic
Most evaluations report a percentage; the few that report a score or a count are mapped to the same 0 to 100 scale or shown unscored. We map each value to a score on which higher means safer, taking the value itself when a high number is the good outcome and 100 minus the value when a low number is. A sub-category score is the mean of its evaluations, a pillar score the mean of its sub-categories, and the index the equal-weight mean of the pillars, in each case over what has been measured. We never fill a missing sub-category with a guess.
A model receives a headline index only when at least 4 of the five pillars have at least one counted measurement. Below that threshold the model keeps its page and its pillar scores but is listed as partial, because averages over two pillars and over five are not comparable. As coverage grows, the rule tightens to require at least half of each counted pillar's sub-categories.
What counts
- One headline measurement per model and evaluation. When several sources report the same evaluation, we take our own run first, then an independent evaluator, then nothing: the developer's own figure is shown on the model page but never enters the index.
- Values read off a chart rather than printed as a number are shown as approximate and not counted.
- Every harm evaluation runs with its benign twin where one exists, so a model cannot look safe by refusing everything. The twin is shown beside the harm row and does not count twice.
- Evaluations that nearly every frontier model now passes close to the ceiling are retired from the mean and kept as legacy rows (StrongREJECT (direct), AgentDojo (static injection), XSTest safe-prompt refusals, BBQ, TruthfulQA MC1, HarmBench (direct requests), Sycophancy: are you sure?, XSTest (HELM Safety)). A retired evaluation still catches a model that is far behind, but it no longer separates the frontier.
- Tests of dangerous knowledge and capability, such as the WMDP questions or the Google DeepMind stealth challenges, are shown on model pages but not scored, since a model that knows more is not a safer model.
- The evaluation-integrity profile (evaluation awareness, sandbagging, alignment faking) is reported beside each model's scores and qualifies them; it is not averaged in.
- Each model is scored on one deployed variant. Where a source distinguishes reasoning-effort settings, we take the default or highest-effort variant and record the alternative readings on the model page.
How SAI runs its own evaluations
Proprietary models are queried through their public APIs with provider-default sampling, reasoning left on, and no system prompt. Open-weight models are served locally with vLLM in bf16 under the same conditions. We follow each evaluation's published protocol and use its published grader, the automated judge that scores each answer; where the original grader is no longer available we name the substitute on the evaluation's page and keep the published prompt template unchanged. Confidence intervals are Wilson intervals from the item counts.
Sometimes a provider's own safety filter blocks a request before the model ever answers it. We report such measurements two ways, as deployed, where the block counts as a refusal, and model only, where the blocked requests are left out. The headline uses the as-deployed reading, because that is what a user of the service gets, and the model page shows the other.
Independent evaluators used in this preview
- Artificial Analysis
- Artificial Analysis (via BenchLM mirror)
- Giskard Phare leaderboard
- Gray Swan AI IPI Arena (with UK AISI / US CAISI)
- MLCommons AILuminate Safety 1.2.1
- Scale AI Fortress leaderboard
- Scale AI SEAL / CAIS MASK leaderboard
- Stanford CRFM HELM AIR-Bench v1.19.0
- Stanford CRFM HELM Safety v1.17.0
- Vectara Hallucination Leaderboard (HHEM-2.3)
These boards differ from one another and from our runs in judges, prompt sets and dates, so cells from different evaluators are comparable only approximately. The "SAI-measured only" view on the index page restricts the index to cells from one pipeline. Numbers from other evaluators that do not correspond to an evaluation in the suite are shown on model pages as cross-checks and never counted.
Version history
- v0.2 preview (2026-11-03). Five pillars and 18 sub-categories replace the six flat categories of v0.1. The suite grows to 78 scored evaluations, chosen to cover the safety areas labs report on (harmful content, over-refusal, jailbreaks, prompt injection, multilingual safety, agent misuse, reward hacking, overreach, factuality, sycophancy, deception, alignment audits, manipulation, self-harm, child safety, bias, privacy, elections). Saturated anchors retire from the mean. An evaluation-integrity profile is added. Pillar 4 becomes "Interaction and honesty" and takes manipulation of the user; privacy splits into memorisation and surveillance assistance.
- v0.1 preview (2026-11-02). First public draft. Six equal-weight categories. SAI runs cover StrongREJECT, AIR-Bench hate and harassment, the XSTest and OR-Bench over-refusal set, AgentHarm, AgentDojo, InstrumentalEval, HealthBench emergency referrals, FACTS Grounding, SimpleQA Verified and AA-Omniscience on three frontier models, and WMDP, TruthfulQA, BBQ and StrongREJECT on open-weight models served locally.