Safety IndexAlibabaQwen3.8-Max
Qwen3.8-Max
AlibabaProprietaryReleased 2026-09Reasoning modelVariant scored: provider default (reasoning on)
Summary
Qwen3.8-Max scores 74 on the SAI Safety Index, rank 3 of 3 models with a headline number. Its strongest area is agentic safety at 81, and its weakest is interaction and honesty at 64.
8 measurements count toward the score, 8 of them from SAI's own runs (13 SAI measurements in total, including context metrics). Not yet measured: adversarial robustness.
Pillar scores
How Qwen3.8-Max compares with every other model scored in the pillar, and which sub-categories the score rests on.
Refusal calibration
Does the model refuse what it should and help with what it may?
- Harmful requests94 from 1 evaluation
- Over-refusal60 from 1 evaluation
Rank 25 of 39 scored models.
Adversarial robustness
Do its refusals and instructions hold up when someone tries to get around them?
- Jailbreaksnot measured
- Prompt injectionnot measured
- Multilingual safetynot measured
9 models scored.
Agentic safety
Does it behave safely when it has tools and autonomy?
- Agent misuse81 from 1 evaluation
- Reward hacking and task integritynot measured
- Overreach and destructive actionsnot measured
Rank 2 of 3 scored models.
Interaction and honesty
Is it honest with the person it is talking to?
- Factuality and hallucination71 from 3 evaluations
- Honesty under pressure and sycophancynot measured
- Deception and scheming57 from 1 evaluation
- Alignment auditnot measured
- Manipulation of the usernot measured
Rank 16 of 29 scored models.
People and society
Does it protect vulnerable people and the public interest?
- Self-harm and wellbeing75 from 1 evaluation
- Child safetynot measured
- Bias, fairness and neutralitynot measured
- Privacy: memorisation and leakagenot measured
- Privacy: surveillance assistancenot measured
- Elections and influence operationsnot measured
Rank 1 of 3 scored models.
Every measurement
These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.
| Evaluation | Value | Score | Source | Setting |
|---|---|---|---|---|
| Refusal calibration | ||||
| Harmful requests | ||||
| AIR-Bench 2024 | 93.9%95% CI 91.7% to 95.5%n = 635 | 94 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks |
| StrongREJECT (direct, violation rate) | 1.3%95% CI 0.5% to 3.2%n = 313 | benign twin, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks |
| StrongREJECT (direct) | 98.7%95% CI 96.8% to 99.5%n = 313 | retired, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); no provider blocks |
| Over-refusal | ||||
| OR-Bench-Hard-1K | 60.2%95% CI 57.7% to 62.6%n = 1,569 | 60 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks |
| Harmless-prompt refusals | 39.8%95% CI 37.4% to 42.3%n = 1,569 | benign twin, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks |
| Adversarial robustness | ||||
| Prompt injection | ||||
| AgentDojo (static injection) | 98.1%95% CI 97.1% to 98.8%n = 1,014 | retired, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; no provider blocks |
| Agentic safety | ||||
| Agent misuse | ||||
| AgentHarm (harmful tasks, safe action rate) | 81.1%n = 176 | 81 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks |
| AgentHarm (benign tasks, helpfulness) | 72.7%n = 176 | benign twin, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks |
| Interaction and honesty | ||||
| Factuality and hallucination | ||||
| AA-Omniscience (public subset) | 25.9%95% CI 22.1% to 30.4%n = 600 | 74 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling |
| FACTS Grounding | 89.8%n = 860 | 90 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationapproximate 2-judge panel gpt-4.1 + google/gemini-2.5-flash (not Google's panel); released v1 json templates; 860 public examples; provider-default sampling, reasoning on |
| SimpleQA Verified | 52.2%95% CI 49.3% to 55%n = 1,000 | 48 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1 |
| AA-Omniscience (public subset) | 27.1%n = 6,000 | read off a chart, not counted | Artificial Analysis (via BenchLM mirror) Independent evaluator · 2026-10-29 | Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'Qwen3.8 Max Preview' |
| Deception and scheming | ||||
| InstrumentalEval | 43.2%95% CI 32.6% to 54.6%n = 74 | 57 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks |
| People and society | ||||
| Self-harm and wellbeing | ||||
| HealthBench emergency referrals | 74.6%95% CI 72.3% to 76.7%n = 482 | 75 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI |