SAI LabsBook a demo

Safety IndexxAIGrok 4

Grok 4

xAIProprietaryReleased 2025-07Reasoning model

SAI Safety Index59provisionalHas 3 of the 4 pillars needed
Strongest pillar97Adversarial robustness
Weakest pillar36Interaction and honesty
Coverage3of 5 pillars3 counted measurements, 0 by SAI

Summary

Grok 4 has data in 3 of the five pillars, short of the 4 needed for a headline number, so it is listed as partly measured; the mean of its measured pillars is 59. Its strongest area is adversarial robustness at 97, and its weakest is interaction and honesty at 36.

3 measurements count toward the score, 0 of them from SAI's own runs. Not yet measured: agentic safety, people and society.

Pillar scores

How Grok 4 compares with every other model scored in the pillar, and which sub-categories the score rests on.

Refusal calibration

Does the model refuse what it should and help with what it may?

  • Harmful requests44 from 1 evaluation
  • Over-refusalnot measured

Rank 37 of 39 scored models.

Adversarial robustness

Do its refusals and instructions hold up when someone tries to get around them?

  • Jailbreaksnot measured
  • Prompt injection97 from 1 evaluation
  • Multilingual safetynot measured

Rank 6 of 9 scored models.

Agentic safety

Does it behave safely when it has tools and autonomy?

Not measured for Grok 4 yet.
  • Agent misusenot measured
  • Reward hacking and task integritynot measured
  • Overreach and destructive actionsnot measured

3 models scored.

Interaction and honesty

Is it honest with the person it is talking to?

  • Factuality and hallucination36 from 1 evaluation
  • Honesty under pressure and sycophancynot measured
  • Deception and schemingnot measured
  • Alignment auditnot measured
  • Manipulation of the usernot measured

Rank 28 of 29 scored models.

People and society

Does it protect vulnerable people and the public interest?

Not measured for Grok 4 yet.
  • Self-harm and wellbeingnot measured
  • Child safetynot measured
  • Bias, fairness and neutralitynot measured
  • Privacy: memorisation and leakagenot measured
  • Privacy: surveillance assistancenot measured
  • Elections and influence operationsnot measured

3 models scored.

Every measurement

These are the raw numbers behind the scores, with who measured them, when, and how. Rows marked counted feed the index. The others are shown for context, such as benign twins of harm rows, retired evaluations that every frontier model passes, tests of dangerous knowledge, other evaluators' overall scores, and the developer's own figures.

EvaluationValueScoreSourceSetting
Refusal calibration
Harmful requests
AIR-Bench 2024% refused (safe) · Higher is better44.4%n = 5,69444Stanford CRFM HELM AIR-Bench v1.19.0
Independent evaluator
Published by Stanford CRFM HELM AIR-Bench v1.19.0.
Full configurationfull 5,694-prompt set, Stanford HELM AIR-Bench v1.19.0; overall refusal rate, 5,694 prompts, GPT-4o judge; HELM row 'Grok 4'
HarmBench (direct requests)safety score 0-100 (HELM) · Higher is better39.7%retired, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety score 0-100, higher is safer (not an attack-success rate); HELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Over-refusal
XSTest (HELM Safety)HELM score 0-100 · Higher is better96.6%retired, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Adversarial robustness
Prompt injection
Gray Swan IPI Arena kit% attack success · Lower is better2.9%n = 12,52697Gray Swan AI IPI Arena (with UK AISI / US CAISI)
Independent evaluator · 2026-03-16
Published by Gray Swan AI IPI Arena (with UK AISI / US CAISI) (2026-03-16).
Full configurationGray Swan IPI Arena, crowdsourced attackers, thinking disabled; indirect prompt injection on agents, breaks/attempts; row 'Grok 4'; arena runs with extended thinking disabled, so this reading may differ from the thinking variant scored elsewhere
Interaction and honesty
Factuality and hallucination
AA-Omniscience (public subset)% hallucination rate · Lower is better64%n = 6,00036Artificial Analysis
Independent evaluator · 2025-11-17
Published by Artificial Analysis (2025-11-17).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Grok 4'
People and society
Bias, fairness and neutrality
BBQ% accuracy (ambiguous) · Higher is better93.7%n = 1,000retired, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Other evaluators' scores (not counted)
Anthropic Red Team prompts (HELM Safety)0-100 · Higher is better95.7%context, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
HELM Safety mean score0-100 · Higher is better83.6%context, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Phare average safety% · Higher is better62.1%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Grok 4'
Phare bias resistance% · Higher is better31.2%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Grok 4'
Phare hallucination resistance% · Higher is better80.7%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Grok 4'
Phare harmfulness resistance% · Higher is better71.8%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Grok 4'
Phare jailbreak resistance% · Higher is better64.8%context, not countedGiskard Phare leaderboard
Independent evaluator · 2026-07-09
Published by Giskard Phare leaderboard (2026-07-09).
Full configurationPhare row 'Grok 4'
SimpleSafetyTests (HELM Safety)0-100 · Higher is better92.2%context, not countedStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'