SAI LabsBook a demo

Safety IndexAlibabaQwen3.8-Max

Qwen3.8-Max

AlibabaProprietaryReleased 2026-09Reasoning modelVariant scored: provider default (reasoning on)

SAI Safety Index#3 of 3743 models ranked so far
Strongest pillar81Agentic safety
Weakest pillar64Interaction and honesty
Coverage4of 5 pillars8 counted measurements, 8 by SAI

Summary

Qwen3.8-Max scores 74 on the SAI Safety Index, rank 3 of 3 models with a headline number. Its strongest area is agentic safety at 81, and its weakest is interaction and honesty at 64.

8 measurements count toward the score, 8 of them from SAI's own runs (13 SAI measurements in total, including context metrics). Not yet measured: adversarial robustness.

Pillar scores

How Qwen3.8-Max compares with every other model scored in the pillar, and which sub-categories the score rests on.

Refusal calibration

Does the model refuse what it should and help with what it may?

  • Harmful requests94 from 1 evaluation
  • Over-refusal60 from 1 evaluation

Rank 25 of 39 scored models.

Adversarial robustness

Do its refusals and instructions hold up when someone tries to get around them?

Not measured for Qwen3.8-Max yet.
  • Jailbreaksnot measured
  • Prompt injectionnot measured
  • Multilingual safetynot measured

9 models scored.

Agentic safety

Does it behave safely when it has tools and autonomy?

  • Agent misuse81 from 1 evaluation
  • Reward hacking and task integritynot measured
  • Overreach and destructive actionsnot measured

Rank 2 of 3 scored models.

Interaction and honesty

Is it honest with the person it is talking to?

  • Factuality and hallucination71 from 3 evaluations
  • Honesty under pressure and sycophancynot measured
  • Deception and scheming57 from 1 evaluation
  • Alignment auditnot measured
  • Manipulation of the usernot measured

Rank 16 of 29 scored models.

People and society

Does it protect vulnerable people and the public interest?

  • Self-harm and wellbeing75 from 1 evaluation
  • Child safetynot measured
  • Bias, fairness and neutralitynot measured
  • Privacy: memorisation and leakagenot measured
  • Privacy: surveillance assistancenot measured
  • Elections and influence operationsnot measured

Rank 1 of 3 scored models.

Every measurement

These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.

EvaluationValueScoreSourceSetting
Refusal calibration
Harmful requests
AIR-Bench 2024% refused (safe) · Higher is better93.9%95% CI 91.7% to 95.5%n = 63594SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks
StrongREJECT (direct, violation rate)% harmful · Lower is better1.3%95% CI 0.5% to 3.2%n = 313benign twin, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks
StrongREJECT (direct)% safe responses · Higher is better98.7%95% CI 96.8% to 99.5%n = 313retired, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks
Over-refusal
OR-Bench-Hard-1K% helpful (not refused) · Higher is better60.2%95% CI 57.7% to 62.6%n = 1,56960SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks
Harmless-prompt refusals% refused · Lower is better39.8%95% CI 37.4% to 42.3%n = 1,569benign twin, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks
Adversarial robustness
Prompt injection
AgentDojo (static injection)% survival · Higher is better98.1%95% CI 97.1% to 98.8%n = 1,014retired, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; no provider blocks
Agentic safety
Agent misuse
AgentHarm (harmful tasks, safe action rate)% safe actions · Higher is better81.1%n = 17681SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks
AgentHarm (benign tasks, helpfulness)% benign tasks completed · Higher is better72.7%n = 176benign twin, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks
Interaction and honesty
Factuality and hallucination
AA-Omniscience (public subset)% hallucination rate · Lower is better25.9%95% CI 22.1% to 30.4%n = 60074SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling
FACTS Grounding% grounded · Higher is better89.8%n = 86090SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationapproximate 2-judge panel gpt-4.1 + google/gemini-2.5-flash (not Google's panel); released v1 json templates; 860 public examples; provider-default sampling, reasoning on
SimpleQA Verified% incorrect · Lower is better52.2%95% CI 49.3% to 55%n = 1,00048SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1
AA-Omniscience (public subset)% hallucination rate · Lower is better27.1%n = 6,000read off a chart, not countedArtificial Analysis (via BenchLM mirror)
Independent evaluator · 2026-10-29
Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'Qwen3.8 Max Preview'
Deception and scheming
InstrumentalEval% instrumental · Lower is better43.2%95% CI 32.6% to 54.6%n = 7457SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks
People and society
Self-harm and wellbeing
HealthBench emergency referralsrubric score % · Higher is better74.6%95% CI 72.3% to 76.7%n = 48275SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI