Safety IndexMoonshot AIKimi K3.1 Pro
Kimi K3.1 Pro
Moonshot AIProprietaryReleased 2026-10Reasoning modelVariant scored: thinking (always on, max effort)
Summary
Kimi K3.1 Pro scores 80 on the SAI Safety Index, rank 1 of 3 models with a headline number. Its strongest area is refusal calibration at 89, and its weakest is people and society at 73.
6 measurements count toward the score, 6 of them from SAI's own runs (11 SAI measurements in total, including context metrics). Not yet measured: adversarial robustness.
Pillar scores
How Kimi K3.1 Pro compares with every other model scored in the pillar, and which sub-categories the score rests on.
Refusal calibration
Does the model refuse what it should and help with what it may?
- Harmful requests94 from 1 evaluation
- Over-refusal84 from 1 evaluation
Rank 7 of 39 scored models.
Adversarial robustness
Do its refusals and instructions hold up when someone tries to get around them?
- Jailbreaksnot measured
- Prompt injectionnot measured
- Multilingual safetynot measured
9 models scored.
Agentic safety
Does it behave safely when it has tools and autonomy?
- Agent misuse78 from 1 evaluation
- Reward hacking and task integritynot measured
- Overreach and destructive actionsnot measured
Rank 3 of 3 scored models.
Interaction and honesty
Is it honest with the person it is talking to?
- Factuality and hallucination72 from 1 evaluation
- Honesty under pressure and sycophancynot measured
- Deception and scheming91 from 1 evaluation
- Alignment auditnot measured
- Manipulation of the usernot measured
Rank 9 of 29 scored models.
People and society
Does it protect vulnerable people and the public interest?
- Self-harm and wellbeing73 from 1 evaluation
- Child safetynot measured
- Bias, fairness and neutralitynot measured
- Privacy: memorisation and leakagenot measured
- Privacy: surveillance assistancenot measured
- Elections and influence operationsnot measured
Rank 2 of 3 scored models.
Every measurement
These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.
| Evaluation | Value | Score | Source | Setting |
|---|---|---|---|---|
| Refusal calibration | ||||
| Harmful requests | ||||
| AIR-Bench 2024 | 93.9%95% CI 91.7% to 95.5%n = 635 | 94 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationhate and harassment slice, 636 of 5,694 prompts; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (paper: GPT-4o-2024-05-13); candidate cap 32,768 tokens; filtered items re-judged via OpenAI-pinned gpt-4o-2024-11-20; no provider blocks |
| StrongREJECT (direct, violation rate) | 0.6%95% CI 0.2% to 2.3%n = 313without provider blocks: 0.6% | benign twin, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal) |
| StrongREJECT (direct) | 99.4%95% CI 97.7% to 99.8%n = 313without provider blocks: 99.4% | retired, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (paper: GPT-4o-mini); Azure-filtered items re-judged via OpenRouter pinned to OpenAI (openai/gpt-4.1); as deployed (a provider block counts as a refusal) |
| Over-refusal | ||||
| OR-Bench-Hard-1K | 84%95% CI 82.1% to 85.7%n = 1,569 | 84 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; helpfulness rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks |
| Harmless-prompt refusals | 16%95% CI 14.3% to 17.9%n = 1,569 | benign twin, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationXSTest-safe (250) + OR-Bench-Hard-1K (1,319) combined set, 1,569 prompts; refusal rate; OpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (originals GPT-4 / GPT-4-Turbo-2024-04-09); temperature 0; no provider blocks |
| Adversarial robustness | ||||
| Prompt injection | ||||
| AgentDojo (static injection) | 100%95% CI 99.6% to 100%n = 1,014 | retired, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by none.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge none; no provider blocks |
| Agentic safety | ||||
| Agent misuse | ||||
| AgentHarm (harmful tasks, safe action rate) | 77.9%n = 176 | 78 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks |
| AgentHarm (benign tasks, helpfulness) | 80%n = 176 | benign twin, not counted | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 refusal + semantic judges (original GPT-4o-2024-08-06); official grading functions; public test split; no provider blocks |
| Interaction and honesty | ||||
| Factuality and hallucination | ||||
| AA-Omniscience (public subset) | 28.1%95% CI 22.8% to 33.7%n = 600 | 72 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling; thinking always on at default max effort |
| AA-Omniscience (public subset) | 44%n = 6,000 | 56duplicate | Artificial Analysis Independent evaluator · 2026-10-23 | Published by Artificial Analysis (2026-10-23).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; Kimi K3.1 Pro launch article |
| Deception and scheming | ||||
| InstrumentalEval | 9.2%95% CI 4.5% to 17.8%n = 76 | 91 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4o 2024-11-20.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4o 2024-11-20 (original GPT-4o); markdown-tolerant re-parse of verdicts; no provider blocks |
| People and society | ||||
| Self-harm and wellbeing | ||||
| HealthBench emergency referrals | 73.1%95% CI 71% to 75.3%n = 482 | 73 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI |