SAI LabsBook a demo

Safety IndexDeepSeekDeepSeek R1

DeepSeek R1

DeepSeekOpen weightsReleased 2025-01Reasoning modelVariant scored: original (Jan 2025); 0528 revision in altValues

SAI Safety Index63provisionalHas 2 of the 4 pillars needed
Strongest pillar69Refusal calibration
Weakest pillar57Interaction and honesty
Coverage2of 5 pillars4 counted measurements, 0 by SAI

Summary

DeepSeek R1 has data in 2 of the five pillars, short of the 4 needed for a headline number, so it is listed as partly measured; the mean of its measured pillars is 63. Its strongest area is refusal calibration at 69, and its weakest is interaction and honesty at 57.

4 measurements count toward the score, 0 of them from SAI's own runs. Not yet measured: adversarial robustness, agentic safety, people and society.

Pillar scores

How DeepSeek R1 compares with every other model scored in the pillar, and which sub-categories the score rests on.

Refusal calibration

Does the model refuse what it should and help with what it may?

  • Harmful requests39 from 2 evaluations
  • Over-refusal100 from 1 evaluation

Rank 31 of 39 scored models.

Adversarial robustness

Do its refusals and instructions hold up when someone tries to get around them?

Not measured for DeepSeek R1 yet.
  • Jailbreaksnot measured
  • Prompt injectionnot measured
  • Multilingual safetynot measured

9 models scored.

Agentic safety

Does it behave safely when it has tools and autonomy?

Not measured for DeepSeek R1 yet.
  • Agent misusenot measured
  • Reward hacking and task integritynot measured
  • Overreach and destructive actionsnot measured

3 models scored.

Interaction and honesty

Is it honest with the person it is talking to?

  • Factuality and hallucinationnot measured
  • Honesty under pressure and sycophancy57 from 1 evaluation
  • Deception and schemingnot measured
  • Alignment auditnot measured
  • Manipulation of the usernot measured

Rank 18 of 29 scored models.

People and society

Does it protect vulnerable people and the public interest?

Not measured for DeepSeek R1 yet.
  • Self-harm and wellbeingnot measured
  • Child safetynot measured
  • Bias, fairness and neutralitynot measured
  • Privacy: memorisation and leakagenot measured
  • Privacy: surveillance assistancenot measured
  • Elections and influence operationsnot measured

3 models scored.

Every measurement

These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.

EvaluationValueScoreSourceSetting
Refusal calibration
Harmful requests
AIR-Bench 2024% refused (safe) · Higher is better52.9%n = 5,69453Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'DeepSeek R1'
Scale FORTRESS (risk score)average risk score 0-100 · Lower is better74.4%95% CI 71.7% to 77.1%26Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'DeepSeek R1'
HarmBench (direct requests)safety score 0-100 (HELM) · Higher is better47.1%retired, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'DeepSeek R1'
Over-refusal
Scale FORTRESS (over-refusal)% benign prompts refused · Lower is better0.5%n = 500100Scale AI Fortress leaderboard
Independent evaluator
Published by Scale AI Fortress leaderboard.
Full configurationFortress row 'DeepSeek R1'
XSTest (HELM Safety)HELM score 0-100 · Higher is better95.3%retired, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
Interaction and honesty
Honesty under pressure and sycophancy
MASK% honest · Higher is better57.3%95% CI 54.7% to 59.9%n = 50057Scale AI SEAL / CAIS MASK leaderboard
Independent evaluator
Published by Scale AI SEAL / CAIS MASK leaderboard.
Full configurationScale leaderboard on the private 500-example split, not the public split; MASK private split, 500 examples; honesty score = 1 - P(lie); row 'DeepSeek R1'
People and society
Bias, fairness and neutrality
BBQ% accuracy (ambiguous) · Higher is better96.6%n = 1,000retired, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
Other evaluators' scores (not counted)
Anthropic Red Team prompts (HELM Safety)0-100 · Higher is better96.2%context, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
HELM Safety mean score0-100 · Higher is better86.5%context, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
Phare average safety% · Higher is better58.5%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Deepseek R1 0528'
Phare bias resistance% · Higher is better25.5%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Deepseek R1 0528'
Phare hallucination resistance% · Higher is better72.9%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Deepseek R1 0528'
Phare harmfulness resistance% · Higher is better95.2%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Deepseek R1 0528'
Phare jailbreak resistance% · Higher is better40.7%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Deepseek R1 0528'
SimpleSafetyTests (HELM Safety)0-100 · Higher is better97.5%context, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
Vectara HHEM hallucination rate% · Lower is better11.3%context, not countedVectara Hallucination Leaderboard (HHEM-2.3)
Independent evaluator · 2026-10-22
Published by Vectara Hallucination Leaderboard (HHEM-2.3) (2026-10-22).
Full configurationsummarization of ~1,000 short documents; row 'DeepSeek R1'