SAI LabsBook a demo

SAI LabsBenchmark Safety BenchmarksNovember 2026

Benchmark Safety Benchmarks

Inspired by the post from Jason Wei, we build an overview of safety benchmarks in AI labs. We read the model cards, system cards and safety reports behind 135 model releases from 18 labs in 2026 and recorded every safety test each one reports. Here are the main results.

18 labs135 releases in 2026, 92 with any safety result305 documents read2,730 safety results recorded124 named benchmarks and test suitesUpdated 2026-11-12

  1. 1

    Most reported

    Labs reporting each benchmark: CyberGym (9); ExploitGym (8); Virology Capabilities Test (VCT), ExploitBench and WMDP (6 each); LAB-Bench (5). Seven of the ten most shared test cyber attack or biology skills. 55 benchmarks are shared by two or more labs.

  2. 2

    Lab by benchmark

    Shared benchmarks reported per lab: Meta (33); OpenAI and Anthropic (28 each); Reflection AI (18); Google DeepMind (17). DeepSeek, Moonshot AI, Z.ai and MiniMax report only cyber and biology ones. Alibaba Qwen, ByteDance Seed and Tencent report none.

  3. 3

    Who built them

    Uses of the benchmarks each group built: the Center for AI Safety and UC Berkeley (27 each); OpenAI (22); SecureBio (20); the UK AI Security Institute (17); Scale AI (16). Universities built 31 percent of uses, non-profits 25 percent, and AI labs 22 percent.

  4. 4

    Risks covered

    Risks covered, out of 17: Meta (17); OpenAI, Anthropic and Google DeepMind (16 each); xAI, Amazon and Reflection AI (13 each). Tencent, DeepSeek, Moonshot AI, Z.ai, MiniMax, Alibaba Qwen and ByteDance Seed cover three or fewer.

  5. 5

    Who can check it

    Only 24 percent of results use a named benchmark, 12 percent come from outside testers, and 64 percent are the labs' own tests. Highest named share: NVIDIA (67%); Meta (62%); Microsoft (54%). The labs with the most results have the lowest named share: OpenAI and Anthropic (17% each); Google DeepMind (14%).

  6. 6

    Outside testers

    Labs name 38 outside testing groups. Labs each group tested for: Gray Swan (6); Apollo Research, Irregular, the UK AI Security Institute and the US Center for AI Standards and Innovation (5 each); Faculty, Handshake AI, METR, Scale AI and SecureBio (3 each).

Most reported

55 benchmarks were reported by two or more labs in 2026. Cyber attack tests lead. Tests of harmful requests and jailbreaks are spread over many benchmarks. So each of those has fewer labs.

Most reported safety benchmarks

Bar length and the large number show how many of the 18 labs reported the benchmark in 2026. The small number is how many of their models it was reported for. Click a row for every result and its source.

1UC Berkeley, 2025 · source9/1826 models
2UC Berkeley, 2026 · source8/1821 models
3SecureBio, 2025 · on request · source6/1837 models
4Carnegie Mellon University, 2026 · source6/1826 models
5CAIS and Scale AI, 2024 · source6/1812 models
6FutureHouse, 2024 · source5/1827 models
7Artificial Analysis, 2025 · source5/1820 models
8Gray Swan and UK AISI, 2025 · not released · source5/1820 models
9Irregular, 2025 · not released · source5/1818 models
10Apollo Research, 2024 · not released · source5/1812 models
11UC Berkeley, 2024 · source5/1811 models
12Stanford, 2024 · source5/189 models
13OpenAI, 2026 · source4/1819 models
14Anthropic, 2025 · source4/1811 models
15OpenAI, 2025 · not released · source4/188 models
16CAIS and Scale AI, 2025 · source3/1821 models
17Anthropic, 2025 · on request · source3/1817 models
18Gray Swan · not released · source3/1816 models
19UIUC, 2025 · source3/1812 models
20UIUC and Purdue, 2025 · source3/1811 models
21Scale AI, 2025 · source3/1810 models
22SecureBio, 2025 · on request · source3/189 models
23SecureBio, 2025 · on request · source3/188 models
24Irregular · not released · source3/188 models
25METR, 2025 · not released · source3/186 models

Lab by benchmark

Each row is a benchmark that at least two labs reported in 2026, and each column is a lab. Labs that publish long safety reports fill many cells. Several labs fill only the cyber rows, because cyber results are the only safety numbers in their release posts.

Which lab reports which benchmark

The number in a cell is how many of that lab's 2026 models reported the benchmark. Darker cells mean more models. A dashed cell means the lab reported it only for an older model. Click a cell for the results.

  • Dangerous skills
  • Harmful requests and refusals
  • Attacks on the model
  • Agent behaviour and control
  • Honesty and users' wellbeing
  • Fairness and privacy
  • Before 2026 only

Scroll sideways to see all 18 labs.

BenchmarkOpenAIAnthropicGoogleMetaxAIMicrosoftAmazonMistralReflectionThinking Mach.NVIDIADeepSeekQwenMoonshotZ.aiMiniMaxByteDanceTencent

Who built them

We credit each benchmark to the institutions that led it, and count one use each time a lab reported it in 2026. The Center for AI Safety and UC Berkeley lead, with 27 uses each. Universities built the benchmarks behind 31 percent of uses, non-profits 25 percent, and the AI labs themselves 22 percent.

University 31%Non-profit 25%AI lab 22%Company 18%
  • University, 31%
  • Non-profit, 25%
  • AI lab, 22%
  • Company, 18%
  • Government, 3%
Center for AI SafetyNon-profitVirology Capabilities Test (VCT), WMDP, MASK, Molecular Biology Capabilities Test (MBCT), Human Pathogen Capabilities Test (HPCT), HLE calibration, HarmBench, D-REX, MACHIAVELLI, TextQuests harm2710 benchmarks
UC BerkeleyUniversityCyberGym, ExploitGym, StrongREJECT, SRE-Bench, ExploitGym honeypot, OR-Bench, MACHIAVELLI277 benchmarks
OpenAIAI labCoT controllability (CoT-Control), CoT monitorability, HealthBench, SimpleQA / SimpleQA Verified, MLE-bench, SWE-bench Verified (AI R&D check), MakeMeSay / MakeMePay, TruthfulQA, First-person fairness, MentalHealthBench2210 benchmarks
SecureBioNon-profitVirology Capabilities Test (VCT), Molecular Biology Capabilities Test (MBCT), ABC-Bench, Human Pathogen Capabilities Test (HPCT), BioTIER, World-Class Biology, DNA synthesis screening evasion207 benchmarks
UK AI Security InstituteGovernmentGray Swan ART and IPI Arena, UK AISI cyber ranges, AgentHarm, UK AISI alignment and sabotage tests, ControlArena, b3 (Backbone Breaker), SandboxEscapeBench177 benchmarks
Scale AICompanyWMDP, MASK, FORTRESS, HLE calibration, DistressBench, PropensityBench166 benchmarks
AnthropicAI labPetri (and Bloom), SHADE-Arena, ImpossibleBench, Political even-handedness, Agentic Misalignment, Alignment faking, Sycophancy: are you sure?137 benchmarks
Gray SwanCompanyGray Swan ART and IPI Arena, Gray Swan Shade red teaming, AgentHarm, D-REX, Gray Swan Reward Hacking Bench135 benchmarks
Google DeepMindAI labSimpleQA / SimpleQA Verified, GDM stealth and situational awareness, ECLeKTic, FACTS Grounding, Gram, HelpBench, Terminal-Bench cheating audit, LongFact, RoboHarm and ASIMOV119 benchmarks
Carnegie Mellon UniversityUniversityExploitBench, ImpossibleBench, PolyGuardPrompts, D-REX104 benchmarks
UIUCUniversityCVE-Bench, SEC-bench, RedCode, HarmBench, DTap95 benchmarks
IrregularCompanyCyScenarioBench, Irregular atomic challenges82 benchmarks
MetaAI labCyberSecEval, AIRS-Bench, CIMemories, AbstentionBench, ADeptS-Bench, HalluLens76 benchmarks
Redwood ResearchNon-profitLinuxArena / LinuxBench, ControlArena, Alignment faking, BashArena, Subversion Strategy Eval75 benchmarks
Stanford UniversityUniversityCybench, AIR-Bench 2024, Terminal-Bench73 benchmarks
Artificial AnalysisCompanyAA-Omniscience, Artificial Analysis Cyber Index72 benchmarks
Apollo ResearchNon-profitApollo Research scheming tests, SAD (Situational Awareness Dataset)62 benchmarks
FutureHouseNon-profitLAB-Bench, BixBench62 benchmarks
METRNon-profitMETR autonomy and time-horizon tests, RE-Bench52 benchmarks
University of OxfordUniversityTruthfulQA, SandboxEscapeBench, Subversion Strategy Eval43 benchmarks

A benchmark built by two institutions counts in full for each, so the bars add up to more than the total. The top bar shows the share of uses by the type of the first-named institution. We show the 20 institutions with the most uses, out of 66.

Risks covered

This grid counts every safety result, including the labs' own tests, by the risk it measures. A filled square means the lab reported at least one result on that risk for a 2026 model. The number is how many results.

  • Dangerous skills
  • Harmful requests and refusals
  • Attacks on the model
  • Agent behaviour and control
  • Honesty and users' wellbeing
  • Fairness and privacy

Scroll sideways to see all risks.

Lab2026 models with safety resultsCyber attack skillBio, chem and nuclearAI research skillHarmful requestsOver-refusalJailbreaksPrompt injectionHarmful agent actionsScheming and sabotageNoticing tests, hiding skillsReward hackingHonesty and hallucinationSycophancy and manipulationMental health and self-harmChild safetyBias and fairnessPrivacyRisks covered
OpenAI20 of 2093162381951238276395128321275810916 of 17
Anthropic13 of 13641028185384736591665139831841262716 of 17
Google DeepMind14 of 14181244927166819981110713516 of 17
Meta7 of 826497216714923162147112217 of 17
xAI4 of 61027151011518633113 of 17
Microsoft6 of 62115141317 of 17
Amazon2 of 2693913111225113 of 17
Mistral AI1 of 463221117 of 17
Reflection AI1 of 1391129413111113 of 17
Thinking Machines4 of 45510639155210 of 17
NVIDIA9 of 121511611187 of 17
DeepSeek1 of 6312 of 17
Alibaba Qwen1 of 1521 of 17
Moonshot AI2 of 5722 of 17
Z.ai3 of 8612 of 17
MiniMax2 of 4222 of 17
ByteDance Seed0 of 30 of 17
Tencent2 of 41223 of 17

Counts cover 2026 documents only. ByteDance Seed published no safety result in 2026, so its row is empty.

Who can check it

A result is easier to check when it comes from a named benchmark that others can look up and often run. We split each lab's results three ways: a named benchmark the lab ran, a test run by a named outside group, and the lab's own tests.

  • Named benchmark
  • Run by a named outside group
  • The lab's own tests
OpenAI17%of 979 results
Anthropic17%of 963 results
Google DeepMind14%of 222 results
Meta62%of 207 results
xAI53%of 91 results
Thinking Machines25%of 51 results
Microsoft54%of 46 results
Amazon39%of 44 results
NVIDIA67%of 43 results
Reflection AI41%of 37 results
Mistral AI56%of 16 results
Moonshot AI33%of 9 results
Z.ai71%of 7 results
Tencent0%of 5 results
DeepSeek75%of 4 results
MiniMax25%of 4 results
Alibaba Qwen100%of 2 results

Each bar splits one lab's 2026 safety results into the three kinds. The percentage is the share from named benchmarks. OpenAI and Anthropic publish the most results by far, and most of them come from their own tests. ByteDance Seed is left out because it published no safety result in 2026.

Outside testers

Some labs have outside groups test a model, and publish what those groups found. These are the groups the labs name in their 2026 safety documents. Two are government bodies: the UK AI Security Institute and the US Center for AI Standards and Innovation.

Gray Swan6 labs: Amazon, Anthropic, Google DeepMind, Meta, OpenAI, Reflection AI
Apollo Research5 labs: Anthropic, Google DeepMind, Meta, OpenAI, Thinking Machines
Irregular5 labs: Anthropic, Google DeepMind, Meta, OpenAI, Reflection AI
UK AI Security Institute5 labs: Anthropic, Google DeepMind, Moonshot AI, OpenAI, Reflection AI
US Center for AI Standards and Innovation5 labs: Anthropic, Google DeepMind, Moonshot AI, OpenAI, Reflection AI
Faculty3 labs: Anthropic, Google DeepMind, Meta
Handshake AI3 labs: Meta, Reflection AI, Thinking Machines
METR3 labs: Amazon, Anthropic, OpenAI
Scale AI3 labs: Meta, Thinking Machines, xAI
SecureBio3 labs: Anthropic, Meta, OpenAI
Andon Labs2 labs: Anthropic, Reflection AI
Artificial Analysis2 labs: Mistral AI, Reflection AI
Deloitte2 labs: Anthropic, Meta
Dyno Therapeutics2 labs: Anthropic, OpenAI
Frontier Design2 labs: Anthropic, Meta
Nemesys Insights2 labs: Amazon, Reflection AI
Trajectory Labs2 labs: Anthropic, Google DeepMind

Named by one lab only: 10a Labs (Anthropic), ActiveFence (Amazon), Aymara (Amazon), Casaba Security (Microsoft), Charlemagne Labs (Meta), Chatterbox Labs (Amazon), Dragos (OpenAI), EnkryptAI (Amazon), Epoch AI (Anthropic), FAR.AI (Thinking Machines), Goodhart Labs (Thinking Machines), Gryphon Scientific (OpenAI), Innodata (Amazon), Mercor (Reflection AI), Mozilla (Anthropic), PrismAI (Amazon), Redwood Research (Anthropic), Signature Science (Anthropic), US DOE NNSA (Anthropic), Vals AI (Thinking Machines), pattrn.ai (Google DeepMind).

Lab by lab

OpenAI, Anthropic, Google DeepMind and Meta publish long safety reports for their main releases. The other labs publish short safety sections or none. Each card links the documents we read.

OpenAI20 of 20 models with safety results · 979 results

Every 2026 release has a system card or addendum on OpenAI's Deployment Safety Hub, from 4 to about 175 pages. Most results come from OpenAI's own tests of harmful content, mental health, jailbreaks and prompt injection. Sections on dangerous skills add cyber and biology tests. The cards name outside testers such as SecureBio, Irregular, Apollo Research and the UK and US government AI institutes.

Shared benchmarks reported: CyberGym, ExploitGym, Virology Capabilities Test (VCT), ExploitBench, WMDP, LAB-Bench

Anthropic13 of 13 models with safety results · 963 results

Every 2026 Claude release has a system card of 135 to 317 pages. The largest section is alignment: automated audits, Petri, SHADE-Arena, MASK, reward hacking and test awareness. The cards also cover cyber and biology tests, harmful and harmless requests, bias, and prompt injection tested by Gray Swan.

Shared benchmarks reported: CyberGym, ExploitGym, Virology Capabilities Test (VCT), ExploitBench, WMDP, LAB-Bench

Google DeepMind14 of 14 models with safety results · 222 results

Gemini 4 Argon and Gemini 4 Carbon each have a long Safety and Security Report, reachable only through a link in the model card. The reports cover dangerous-skill tests for cyber, biology and AI research, plus jailbreaks, alignment and prompt injection. Flash, Flash-Lite and Gemma cards carry a short standard safety table. The report for Gemini 4 Hydrogen, released 12 November, is announced but not yet out.

Shared benchmarks reported: ExploitGym, ExploitBench, AA-Omniscience, Gray Swan ART and IPI Arena, CyScenarioBench, Apollo Research scheming tests

Meta7 of 8 models with safety results · 207 results

Muse Spark, Muse Spark 1.1 and Muse Spark 1.3 each have a safety and preparedness report of 110 to 160 pages. They report many public benchmarks: VCT, WMDP, CyberGym, ExploitGym, StrongREJECT, AgentDojo, MASK and SHADE-Arena. Meta released no Llama model in 2026.

Shared benchmarks reported: CyberGym, ExploitGym, Virology Capabilities Test (VCT), ExploitBench, WMDP, LAB-Bench

xAI4 of 6 models with safety results · 91 results

xAI publishes a PDF model card for each Grok flagship. The Grok 4.20 card covers the most risks, including an alignment audit and prompt injection. Later cards reuse a fixed template of cyber, biology, jailbreak and refusal tests, and name few outside testers.

Shared benchmarks reported: CyberGym, Virology Capabilities Test (VCT), WMDP, LAB-Bench, StrongREJECT, Cybench

Microsoft6 of 6 models with safety results · 46 results

Microsoft's MAI models have PDF model cards, and MAI-Thinking-1 has a long technical paper with a safety section. The two cyber models report the most benchmarks, all on cyber attack skill. The paper says dangerous-capability tests were out of scope for MAI-Thinking-1.

Shared benchmarks reported: CyberGym, ExploitGym, CVE-Bench, CyberSecEval, CyberBench (Liu et al.), MakeMeSay / MakeMePay

Amazon2 of 2 models with safety results · 44 results

Amazon's safety results are in its Frontier Model Safety Framework report for Nova 2 Lite, published in January 2026, and in AWS service cards. The report covers biology, chemistry, cyber and AI research tests, with outside reviews by Nemesys Insights and METR. Amazon reports nothing on alignment or scheming.

Shared benchmarks reported: Virology Capabilities Test (VCT), WMDP, LAB-Bench, StrongREJECT, Cybench, METR autonomy and time-horizon tests

Mistral AI1 of 4 models with safety results · 16 results

Only Mistral Large 4 has safety results, in its technical report and launch posts. Most are about cyber: refusals of cyber attack requests and cyber skill benchmarks run by Artificial Analysis. It also reports one prompt injection test (b3) and one child-safety test (KORA). Mistral says a fuller safety report is coming.

Shared benchmarks reported: CyberGym, StrongREJECT, Cybench, AgentHarm, CWE-Bench, Artificial Analysis Cyber Index

Reflection AI1 of 1 models with safety results · 37 results

Reflection's one 2026 release, Beam 1.0, has a 40-page safety section in its technical report. It covers biology tests from SecureBio, a trial of whether the model helps people with lab tasks, cyber tests by Irregular, Petri audits, and prompt injection on AgentDojo. Gray Swan, Andon Labs, Handshake and Mercor tested it from outside.

Shared benchmarks reported: Virology Capabilities Test (VCT), WMDP, AA-Omniscience, Gray Swan ART and IPI Arena, CyScenarioBench, Petri (and Bloom)

Thinking Machines4 of 4 models with safety results · 51 results

Thinking Machines has no system cards. Its model cards and launch posts report FORTRESS and StrongREJECT for every Inkling model. Inkling 2 Small adds cyber, distress and reward-hacking tests. Outside testers (Scale AI, Handshake AI, FAR.AI and Apollo Research) are named, mostly without numbers.

Shared benchmarks reported: Apollo Research scheming tests, StrongREJECT, FORTRESS, HarmBench, DistressBench, HoneyBench

NVIDIA9 of 12 models with safety results · 43 results

NVIDIA's Nemotron model cards include short safety, bias and privacy subcards. They name tests such as BBQ and Aegis but give few scores. The one number on every flagship is AA-Omniscience, used as a hallucination measure.

Shared benchmarks reported: AA-Omniscience, BBQ, TruthfulQA, Aegis, garak, HalluLens

DeepSeek1 of 6 models with safety results · 4 results

DeepSeek's 2026 model cards and technical reports have no safety section. Its 2026 risk results are all in the V4.1-Flash report: three cyber benchmarks and a short account of agents cheating during tests. Its last safety report was the DeepSeek-R1 supplement of September 2025.

Shared benchmarks reported: CyberGym, ExploitGym, SEC-bench

Alibaba Qwen1 of 15 models with safety results · 2 results

Qwen's 2026 launch posts and model cards show only capability tables. The one exception is a cybersecurity appendix in the Qwen3-Coder-Next report. Its last safety report was the Qwen3Guard report of October 2025.

Shared benchmarks reported: CWEval, SecCodeBench

Moonshot AI2 of 5 models with safety results · 9 results

Most of Moonshot's results are about cyber. The Kimi K3 report has a cyber section and cites a joint test by the UK and US government AI institutes. The K3.1 Pro card reports ExploitBench and ExploitGym. Both models also report an internal hallucination rate.

Shared benchmarks reported: ExploitGym, ExploitBench, UK AISI cyber ranges

Z.ai3 of 8 models with safety results · 7 results

Z.ai publishes no system cards or safety reports. Its scored risk results are all cyber attack benchmarks: CyberGym for three GLM releases, and ExploitBench and ExploitGym for GLM-5.3. It delayed GLM-5.3's open weights for a safety check but did not publish the results.

Shared benchmarks reported: CyberGym, ExploitGym, ExploitBench

MiniMax2 of 4 models with safety results · 4 results

MiniMax's 2026 launch posts and model cards contain only capability tables. Its 2026 risk results are a CyberGym score for M3.1 Flash, presented as a skill, and checks that its models did not cheat on one coding benchmark. MiniMax names no safety framework or outside testers.

Shared benchmarks reported: CyberGym

ByteDance Seed0 of 3 models with safety results · 0 results

ByteDance Seed's 2026 model cards have no safety section. Its latest safety results are AIR-Bench and XSTest charts in the Seed1.8 card from December 2025.

Tencent2 of 4 models with safety results · 5 results

Tencent's Hy3 post reports an internal hallucination rate. The Hy4 launch post, from 4 November, is its first with a safety section. It reports an internal biology test (30.4 out of 100, below Tencent's risk line) and a cyber red team without scores. Tencent says a full system card will come with the open weights.

How we counted

We covered 18 labs: the twelve on Jason Wei's benchmark-of-benchmarks count plus Microsoft, Amazon, Mistral AI, MiniMax, ByteDance Seed and Tencent. For each lab, a research agent found every language, reasoning and agent model it released between 1 January and 12 November 2026. It then read the launch post, the model or system card, and any separate safety report, including PDFs linked from inside a model card. Every result on this page links to the page or PDF it came from.

A safety result is any number or named test a lab reports to measure harm or risk. That covers:

  • dangerous skills, such as cyber attacks, biology and chemistry, and AI research;
  • harmful requests, over-refusal, jailbreaks and prompt injection;
  • scheming, sabotage, test awareness and reward hacking;
  • honesty, sycophancy, mental health, child safety, bias and privacy.

We counted cyber attack benchmarks even when a lab presents them as a skill rather than a risk. The SAI Safety Index treats cyber attack skill as a risk, and several labs report the same benchmark both ways. We left out scores of separate safety classifiers, model welfare assessments, and research papers that were not part of a model release.

One result is one test reported for one release. A lab that reports the same test for five models counts five results but one lab. The ranking counts labs, so a lab with many releases does not outweigh a lab with one. We merged versions and subsets of one benchmark. For example, WMDP-Bio and WMDP-Chem count as WMDP. We kept different benchmarks with similar names apart.

We count a result in 2026 when its document came out in 2026. So Amazon's Nova 2 safety report counts, and we count the Nova 2 family as a 2026 release. The models came out in December 2025, but the report came out in January 2026. DeepSeek, Alibaba Qwen, MiniMax and ByteDance Seed published little or no safety work in 2026. For these labs we also read their latest older safety document. Its results appear as “before 2026” in the lab-by-benchmark grid and the lab cards, and are not counted anywhere else.

These are the limits of the count:

  • Many results appear only in charts. We recorded the test and any number stated in the text, so some results show no score.
  • Some pages block automated reading, including openai.com and x.ai. For those labs we read the system cards from their safety hubs and found the release list through search.
  • A lab's own test may be the same as another lab's under a different name. We could only merge tests whose names match.
  • Gemini 4 Hydrogen (released 12 November) and Muse Wonder have safety reports announced but not yet published.