SAI LabsBook a demo

Safety IndexEvaluationsAA-Omniscience (public subset)

AA-Omniscience (public subset)

Interaction and honesty · Factuality and hallucinationCounts in the indexRun by SAI

Of the questions the model does not get right, how many it answers confidently wrong instead of admitting it does not know.

Metric
% hallucination rate. Lower is better.
Dataset
huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public · 300-600 public (3,000 full v1.1) · Apache-2.0
Grader
Gemini 2.5 Flash (paper's grader); AA now uses GPT-5.6 Luna (the paper used Gemini 2.5 Flash)
Status
Run by SAI. Measured on 15 models, 5 readings by SAI.
Replicability
public
Notes
Two readings are kept separate: the SAI run on the 600 public questions and Artificial Analysis's own result on the full set.

Results

ModelValueScoreSourceSetting
Gemini 4 Argon4.0%n = 6,00096Artificial Analysis
Independent evaluator · 2026-11-02 (fetched)
Published by Artificial Analysis (2026-11-02 (fetched)).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'Gemini 4 Argon (high)'
Gemini 4 Carbon11%n = 6,00089Artificial Analysis
Independent evaluator · 2026-10-28
Published by Artificial Analysis (2026-10-28).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'Gemini 4 Carbon (high)'
Gemini 4 Argon24.5%95% CI 19% to 29.5%n = 60076SAI
Measured by SAI · 2026-10-23
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling
Qwen3.8-Max25.9%95% CI 22.1% to 30.4%n = 60074SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling
Claude Haiku 4.526%n = 6,00074Artificial Analysis
Independent evaluator · 2025-11-16
Published by Artificial Analysis (2025-11-16).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Claude Haiku 4.5'
Kimi K3.1 Pro28.1%95% CI 22.8% to 33.7%n = 60072SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling; thinking always on at default max effort
Kimi K333.2%95% CI 28.3% to 38.4%n = 60067SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling
Kimi K3.1 Pro44%n = 6,00056Artificial Analysis
Independent evaluator · 2026-10-23
Published by Artificial Analysis (2026-10-23).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; Kimi K3.1 Pro launch article
Claude Sonnet 4.548%n = 6,00052Artificial Analysis
Independent evaluator · 2025-11-16
Published by Artificial Analysis (2025-11-16).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Claude Sonnet 4.5'
Claude Opus 5.550%n = 6,00050Artificial Analysis
Independent evaluator · 2026-10-28
Published by Artificial Analysis (2026-10-28).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'Claude Opus 5.5 (max)'
GPT-6 Astra50%n = 6,00050Artificial Analysis
Independent evaluator · 2026-10-28
Published by Artificial Analysis (2026-10-28).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'GPT-6 Astra (max)'
Kimi K353%n = 6,00047Artificial Analysis
Independent evaluator · 2026-10-23
Published by Artificial Analysis (2026-10-23).
Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; Kimi K3.1 Pro launch article
Claude Opus 4.558%n = 6,00042Artificial Analysis
Independent evaluator · 2025-11-25
Published by Artificial Analysis (2025-11-25).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Claude Opus 4.5 (thinking)'
Grok 464%n = 6,00036Artificial Analysis
Independent evaluator · 2025-11-17
Published by Artificial Analysis (2025-11-17).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Grok 4'
GPT-581%n = 6,00019Artificial Analysis
Independent evaluator · 2025-11-17
Published by Artificial Analysis (2025-11-17).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'GPT-5 (high)'
Qwen3.5-9B99.2%95% CI 98.5% to 99.8%n = 6001SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling; REASONING DISABLED (--no-reasoning)
GLM-5.226.3%n = 6,000read off a chartArtificial Analysis (via BenchLM mirror)
Independent evaluator · 2026-10-29
Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'GLM-5.2'
Grok 4.325%n = 6,000read off a chartArtificial Analysis (via BenchLM mirror)
Independent evaluator · 2026-10-29
Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'Grok 4.3'
Qwen3.8-Max27.1%n = 6,000read off a chartArtificial Analysis (via BenchLM mirror)
Independent evaluator · 2026-10-29
Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).
Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'Qwen3.8 Max Preview'