SAI LabsBook a demo

Safety IndexMoonshot AIKimi K3.1 Pro

Kimi K3.1 Pro

Moonshot AIProprietaryReleased 2026-10Reasoning modelVariant scored: thinking (always on, max effort)

SAI Safety Index#1 of 3803 models ranked so far
Strongest pillar89Refusal calibration
Weakest pillar73People and society
Coverage4of 5 pillars6 counted measurements, 6 by SAI

Summary

Kimi K3.1 Pro scores 80 on the SAI Safety Index, rank 1 of 3 models with a headline number. Its strongest area is refusal calibration at 89, and its weakest is people and society at 73.

6 measurements count toward the score, 6 of them from SAI's own runs (11 SAI measurements in total, including context metrics). Not yet measured: adversarial robustness.

Pillar scores

How Kimi K3.1 Pro compares with every other model scored in the pillar, and which sub-categories the score rests on.

Refusal calibration

Does the model refuse what it should and help with what it may?

  • Harmful requests94 from 1 evaluation
  • Over-refusal84 from 1 evaluation

Rank 7 of 39 scored models.

Adversarial robustness

Do its refusals and instructions hold up when someone tries to get around them?

Not measured for Kimi K3.1 Pro yet.
  • Jailbreaksnot measured
  • Prompt injectionnot measured
  • Multilingual safetynot measured

9 models scored.

Agentic safety

Does it behave safely when it has tools and autonomy?

  • Agent misuse78 from 1 evaluation
  • Reward hacking and task integritynot measured
  • Overreach and destructive actionsnot measured

Rank 3 of 3 scored models.

Interaction and honesty

Is it honest with the person it is talking to?

  • Factuality and hallucination72 from 1 evaluation
  • Honesty under pressure and sycophancynot measured
  • Deception and scheming91 from 1 evaluation
  • Alignment auditnot measured
  • Manipulation of the usernot measured

Rank 9 of 29 scored models.

People and society

Does it protect vulnerable people and the public interest?

  • Self-harm and wellbeing73 from 1 evaluation
  • Child safetynot measured
  • Bias, fairness and neutralitynot measured
  • Privacy: memorisation and leakagenot measured
  • Privacy: surveillance assistancenot measured
  • Elections and influence operationsnot measured

Rank 2 of 3 scored models.

Every measurement

These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.

EvaluationValueScoreSourceSetting
Refusal calibration
Harmful requests
AIR-Bench 2024% refused (safe) · Higher is better93.9%95% CI 91.7% to 95.5%n = 63594SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks
StrongREJECT (direct, violation rate)% harmful · Lower is better0.6%95% CI 0.2% to 2.3%n = 313without provider blocks: 0.6%benign twin, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal)
StrongREJECT (direct)% safe responses · Higher is better99.4%95% CI 97.7% to 99.8%n = 313without provider blocks: 99.4%retired, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal)
Over-refusal
OR-Bench-Hard-1K% helpful (not refused) · Higher is better84%95% CI 82.1% to 85.7%n = 1,56984SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks
Harmless-prompt refusals% refused · Lower is better16%95% CI 14.3% to 17.9%n = 1,569benign twin, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks
Adversarial robustness
Prompt injection
AgentDojo (static injection)% survival · Higher is better100%95% CI 99.6% to 100%n = 1,014retired, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; no provider blocks
Agentic safety
Agent misuse
AgentHarm (harmful tasks, safe action rate)% safe actions · Higher is better77.9%n = 17678SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks
AgentHarm (benign tasks, helpfulness)% benign tasks completed · Higher is better80%n = 176benign twin, not countedSAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks
Interaction and honesty
Factuality and hallucination
AA-Omniscience (public subset)% hallucination rate · Lower is better28.1%95% CI 22.8% to 33.7%n = 60072SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling; thinking always on at default max effort
AA-Omniscience (public subset)% hallucination rate · Lower is better44%n = 6,00056duplicateArtificial Analysis
Independent evaluator · 2026-10-23
Published by Artificial Analysis (2026-10-23).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; Kimi K3.1 Pro launch article
Deception and scheming
InstrumentalEval% instrumental · Lower is better9.2%95% CI 4.5% to 17.8%n = 7691SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks
People and society
Self-harm and wellbeing
HealthBench emergency referralsrubric score % · Higher is better73.1%95% CI 71% to 75.3%n = 48273SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI