SAI LabsBook a demo

Safety IndexAlibabaQwen3-8B

Qwen3-8B

AlibabaOpen weightsReleased 2025-04Reasoning modelVariant scored: thinking

SAI Safety Index36provisionalHas 1 of the 4 pillars needed
Strongest pillar36Adversarial robustness
Weakest pillar–No data
Coverage1of 5 pillars1 counted measurements, 1 by SAI

Summary

Qwen3-8B has data in 1 of the five pillars, short of the 4 needed for a headline number, so it is listed as partly measured; the mean of its measured pillars is 36. Its strongest area is adversarial robustness at 36.

1 measurements count toward the score, 1 of them from SAI's own runs (5 SAI measurements in total, including context metrics). Not yet measured: refusal calibration, agentic safety, interaction and honesty, people and society.

Pillar scores

How Qwen3-8B compares with every other model scored in the pillar, and which sub-categories the score rests on.

Refusal calibration

Does the model refuse what it should and help with what it may?

Not measured for Qwen3-8B yet.
  • Harmful requestsnot measured
  • Over-refusalnot measured

39 models scored.

Adversarial robustness

Do its refusals and instructions hold up when someone tries to get around them?

  • Jailbreaks36 from 1 evaluation
  • Prompt injectionnot measured
  • Multilingual safetynot measured

Rank 9 of 9 scored models.

Agentic safety

Does it behave safely when it has tools and autonomy?

Not measured for Qwen3-8B yet.
  • Agent misusenot measured
  • Reward hacking and task integritynot measured
  • Overreach and destructive actionsnot measured

3 models scored.

Interaction and honesty

Is it honest with the person it is talking to?

Not measured for Qwen3-8B yet.
  • Factuality and hallucinationnot measured
  • Honesty under pressure and sycophancynot measured
  • Deception and schemingnot measured
  • Alignment auditnot measured
  • Manipulation of the usernot measured

29 models scored.

People and society

Does it protect vulnerable people and the public interest?

Not measured for Qwen3-8B yet.
  • Self-harm and wellbeingnot measured
  • Child safetynot measured
  • Bias, fairness and neutralitynot measured
  • Privacy: memorisation and leakagenot measured
  • Privacy: surveillance assistancenot measured
  • Elections and influence operationsnot measured

3 models scored.

Every measurement

These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.

EvaluationValueScoreSourceSetting
Refusal calibration
Over-refusal
XSTest safe-prompt refusals% refused of 250 safe prompts · Lower is better7.7%n = 233retired, not countedSAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationvLLM, thinking on unless noted, no system prompt; research run; any-refusal rate (full refusal 2.1%); judge azure gpt-4.1-mini
Adversarial robustness
Jailbreaks
StrongREJECT with jailbreak templatesmean harmfulness score x100 under attack · Lower is better64%n = 22736SAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by azure gpt-4.1-mini.
Full configurationAIM template only (one static jailbreak wrapper, not the full template set); vLLM, thinking on unless noted, no system prompt; research run; judge azure gpt-4.1-mini
Interaction and honesty
Factuality and hallucination
TruthfulQA MC1% accuracy · Higher is better70.9%n = 817retired, not countedSAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM, thinking on unless noted, no system prompt; research run; judge none
TruthfulQA MC1% accuracy · Higher is better71.4%n = 817retired, not countedSAI
Measured by SAI · 2026-10-21
Run by SAI on the open-weight model served locally; graded by none.
Full configurationSAI engine, local vLLM serving, thinking on, no system prompt, default sampling; full set; judge none (multiple choice); stderr 1.58 points
Capability context (not scored)
WMDP-Chem% accuracy · Capability, not scored55.5%n = 200context, not countedSAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM, thinking on unless noted, no system prompt; research run; 200-item subset with MMLU-chem baseline 64.5; judge none