SAI LabsBook a demo

SAISafety Indexv0.2 preview

SAI Safety Index

As AI models become more capable, how safely they behave matters more and is getting harder to judge from the labs' own reports. It is thus important to have independent evidence. The SAI Safety Index puts the models on one scale, five pillars of behavioural evaluations, each scored so that higher means safer and averaged into one number per model.

54 models3 with a headline index78 evaluations in the suite: 9 run, 64 built, 5 planned367 measurements from SAI and 9 independent evaluatorsUpdated 2026-11-03

Highest index

Kimi K3.1 Pro leads the SAI Safety Index at 80 among models with enough coverage, followed by Gemini 4 Argon at 75 and Qwen3.8-Max at 74.

Still to come

42 more models are partly measured and wait for runs in the pillars they lack, and 64 of the 78 evaluations in the suite are built and waiting for their first runs. The index fills in as they land.

Index

Each bar is a model's index; hover for the five pillar scores, click for the model's page.

SAI Safety Index

Equal-weight mean of five pillar scores, 0 to 100. Higher is safer. 3 of 54 models have enough coverage for a headline number.

  • Ranked, mostly SAI runs
  • Partly measured, provisional mean, not ranked
1Kimi K3.1 Pro802Gemini 4 Argon753Qwen3.8-Max74
Partly measured: 3 of 5 pillars (7 models). No rank; the bar is the mean of the measured pillars only.
Claude Haiku 4.5893 of 5GPT-5.1883 of 5Claude Sonnet 4.5873 of 5Claude Opus 4.5863 of 5GPT-5783 of 5DeepSeek V3.1603 of 5Grok 4593 of 5
Partly measured: 2 of 5 pillars (15 models). No rank; the bar is the mean of the measured pillars only.
gpt-oss-120b912 of 5
20 more models have data in only one pillar

How the index is built

Five pillars with equal weight, each split into sub-categories that answer one plain question, each measured by one or more public evaluations. A model's sub-category score is the mean of its evaluations, the pillar score the mean of its sub-categories, and the index the mean of its pillars. A model needs data in at least 4 of the five pillars to get a headline number. All 78 evaluations and the full methodology.

PillarSub-categoryEvaluations
Refusal calibrationDoes the model refuse what it should and help with what it may?Harmful requestsDoes the model decline clearly harmful requests?
Over-refusalDoes it still help with benign requests that merely sound risky?
Adversarial robustnessDo its refusals and instructions hold up when someone tries to get around them?JailbreaksDo its refusals survive adversarial prompting?
Prompt injectionDoes it ignore instructions smuggled into documents, web pages and tool outputs?
Multilingual safetyIs it as safe in other languages as it is in English?
Agentic safetyDoes it behave safely when it has tools and autonomy?Agent misuseWhen acting with tools, does it refuse to carry out harmful tasks?
Reward hacking and task integrityDoes it do the task honestly rather than gaming the checks or faking success?
Overreach and destructive actionsDoes it ask before taking risky or irreversible actions, and avoid them when it should?
Interaction and honestyIs it honest with the person it is talking to?Factuality and hallucinationDoes it tell the truth, and admit what it does not know?
Honesty under pressure and sycophancyDoes it hold a correct answer when the user pushes back, and avoid telling people what they want to hear?
Deception and schemingWhen given a reason to, does it deceive, scheme or resist being shut down?
Alignment auditWhat does an open-ended automated audit of its behaviour turn up?
Manipulation of the userDoes it avoid nudging, pressuring or manipulating the person it is talking to?
People and societyDoes it protect vulnerable people and the public interest?Self-harm and wellbeingDoes it respond well to someone who may be in crisis or becoming dependent on it?
Child safetyDoes it keep children safe and refuse to help those who would harm them?
Bias, fairness and neutralityDoes it treat people and political views even-handedly?
Privacy: memorisation and leakageDoes it keep personal data it has seen or been told to itself?
Privacy: surveillance assistanceDoes it refuse to help track, profile or expose people?
Elections and influence operationsDoes it refuse to help with election manipulation and influence campaigns?
  • green measured by SAI
  • dark independent evaluator's number
  • grey no measurement yet; the chip says whether the evaluation is run, built or scheduled

Refusal balance

A model can look safe simply by refusing a lot, so this chart pairs how often it declines genuinely harmful requests with how often it still helps with benign requests that merely sound dangerous. The top-right corner is where both are high.

Refuses harm, still helps

Harmful-requests sub-category score (vertical) against over-refusal sub-category score (horizontal). Higher is better on both axes.

  • Both scores from SAI runs
  • At least one score from an independent evaluator
  • Both above 80
405060708090100405060708090100Over-refusal score (helps with benign requests that sound risky)Harmful-requests score (declines harmful requests)Qwen3.8-MaxKimi K3.1 ProGemini 4 ArgonLlama 4 MaverickDeepSeek R1

Axes start at 40. A point is a model; values are sub-category scores, so a model's position can rest on different evaluations from its neighbour's. Hover a point for the exact numbers.

Pillars

The index is the average of these five. Each card shows the pillar score, the mean of its sub-categories after mapping every metric to a 0 to 100 scale where higher is safer, and how much of the pillar is measured so far.

Refusal calibration

Does the model refuse what it should and help with what it may?

  • Harmful requests2 of 7 evaluations measured, 39 models
  • Over-refusal2 of 5 evaluations measured, 5 models

39 models scored, top 8 shown.

Adversarial robustness

Do its refusals and instructions hold up when someone tries to get around them?

  • Jailbreaks1 of 4 evaluations measured, 2 models
  • Prompt injection1 of 8 evaluations measured, 7 models
  • Multilingual safety0 of 5 evaluations measured

9 models scored, top 8 shown.

Agentic safety

Does it behave safely when it has tools and autonomy?

  • Agent misuse1 of 3 evaluations measured, 3 models
  • Reward hacking and task integrity0 of 5 evaluations measured
  • Overreach and destructive actions0 of 3 evaluations measured

3 models scored, all shown.

Interaction and honesty

Is it honest with the person it is talking to?

  • Factuality and hallucination3 of 5 evaluations measured, 13 models
  • Honesty under pressure and sycophancy1 of 6 evaluations measured, 19 models
  • Deception and scheming1 of 4 evaluations measured, 3 models
  • Alignment audit0 of 2 evaluations measured
  • Manipulation of the user0 of 3 evaluations measured

29 models scored, top 8 shown.

People and society

Does it protect vulnerable people and the public interest?

  • Self-harm and wellbeing1 of 3 evaluations measured, 3 models
  • Child safety0 of 2 evaluations measured
  • Bias, fairness and neutrality0 of 3 evaluations measured
  • Privacy: memorisation and leakage0 of 2 evaluations measured
  • Privacy: surveillance assistance0 of 4 evaluations measured
  • Elections and influence operations0 of 4 evaluations measured

3 models scored, all shown.

What is measured

The index is only as good as its coverage, so this grid shows which models have been measured on which scored evaluations. Each filled cell is one measurement; green ones are SAI runs, dark ones come from independent evaluators, and hatched ones were read off a chart and are shown but not counted. Evaluations that are built but not yet run on any model are listed in the catalogue, not here.

How it works

Every number on this page links to the measurement behind it, with who ran it, when, and under what setting. Where we have run the evaluation ourselves we mark it as SAI-measured, and where we have not yet, we show the best independent measurement we could find and say so.

Every evaluation reports a number in its own units, such as a refusal rate, an attack success rate or an accuracy. We first map each one to a 0 to 100 scale on which higher means safer, flipping the metrics where a low number is the good outcome. Scores are then averaged upward: evaluations into sub-categories, sub-categories into pillars, pillars into the index, each step an equal-weight mean of what has been measured. We never fill a missing sub-category with a guess, so a model with data in four pillars is scored on those four and labelled as such.

Evaluations that every frontier model now passes near the ceiling are retired from the mean and kept on the model pages as legacy rows. Tests of dangerous knowledge and capability are shown as context, never scored. The evaluation-integrity profile, which asks whether a model knows it is being tested, holds back, or fakes alignment, sits beside the scores and qualifies them.

Read the methodology, including the pillar table, the rules for what counts, the automated graders used, and the version history.

Questions

Is a higher index always better?

Yes, every score on this site reads the same way. Metrics where a low number is good, such as an attack success rate or a hallucination rate, are flipped before they enter the index. The raw value and its direction are still shown on each model page.

Why does one model get a number and another not?

A model needs at least 4 of the five pillars covered. Models with less coverage appear in the table as partial and have a model page, but no headline number, because an average of two pillars is not comparable with an average of five.

Why are so many evaluations listed but not yet measured?

Version 0.2 of the suite has 78 evaluations, chosen to cover the safety areas labs report on rather than the few that happen to be easy to run. 9 have been run on at least one model, 64 more are built and waiting for their first runs, and 5 are planned. The catalogue shows each one's status, and the index fills in as the runs land.

Can numbers from different evaluators be averaged?

They can be averaged only with care, and this preview says so wherever it does it. The automated graders, prompt sets and dates differ between evaluators, so two models whose index rests on different sources are not on exactly the same footing. The SAI-measured view shows the subset where every measurement came from our own runs. As our own coverage grows, the borrowed numbers drop out.

How is this different from the labs' own safety reports?

Labs evaluate their own models on prompt sets nobody else can run, with protocols that change between releases. We show those numbers as context on the model pages and never count them in the index. Everything in the index was run by SAI or by an evaluator independent of the model's developer.