Safety IndexEvaluationsAA-Omniscience (public subset)
AA-Omniscience (public subset)
Interaction and honesty · Factuality and hallucinationCounts in the indexRun by SAI
Of the questions the model does not get right, how many it answers confidently wrong instead of admitting it does not know.
- Metric
- % hallucination rate. Lower is better.
- Dataset
- huggingface.co/datasets/ArtificialAnalysis/AA-Omniscience-Public · 300-600 public (3,000 full v1.1) · Apache-2.0
- Grader
- Gemini 2.5 Flash (paper's grader); AA now uses GPT-5.6 Luna (the paper used Gemini 2.5 Flash)
- Status
- Run by SAI. Measured on 15 models, 5 readings by SAI.
- Replicability
- public
- Notes
- Two readings are kept separate: the SAI run on the 600 public questions and Artificial Analysis's own result on the full set.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 4.0%n = 6,000 | 96 | Artificial Analysis Independent evaluator · 2026-11-02 (fetched) | Published by Artificial Analysis (2026-11-02 (fetched)).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'Gemini 4 Argon (high)' | |
| 11%n = 6,000 | 89 | Artificial Analysis Independent evaluator · 2026-10-28 | Published by Artificial Analysis (2026-10-28).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'Gemini 4 Carbon (high)' | |
| 24.5%95% CI 19% to 29.5%n = 600 | 76 | SAI Measured by SAI · 2026-10-23 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling | |
| 25.9%95% CI 22.1% to 30.4%n = 600 | 74 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling | |
| 26%n = 6,000 | 74 | Artificial Analysis Independent evaluator · 2025-11-16 | Published by Artificial Analysis (2025-11-16).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Claude Haiku 4.5' | |
| 28.1%95% CI 22.8% to 33.7%n = 600 | 72 | SAI Measured by SAI · 2026-10-24 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling; thinking always on at default max effort | |
| 33.2%95% CI 28.3% to 38.4%n = 600 | 67 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling | |
| 44%n = 6,000 | 56 | Artificial Analysis Independent evaluator · 2026-10-23 | Published by Artificial Analysis (2026-10-23).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; Kimi K3.1 Pro launch article | |
| 48%n = 6,000 | 52 | Artificial Analysis Independent evaluator · 2025-11-16 | Published by Artificial Analysis (2025-11-16).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Claude Sonnet 4.5' | |
| 50%n = 6,000 | 50 | Artificial Analysis Independent evaluator · 2026-10-28 | Published by Artificial Analysis (2026-10-28).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'Claude Opus 5.5 (max)' | |
| 50%n = 6,000 | 50 | Artificial Analysis Independent evaluator · 2026-10-28 | Published by Artificial Analysis (2026-10-28).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; row 'GPT-6 Astra (max)' | |
| 53%n = 6,000 | 47 | Artificial Analysis Independent evaluator · 2026-10-23 | Published by Artificial Analysis (2026-10-23).Full configurationArtificial Analysis, 3,000 questions; AA-Omniscience v1.1; Kimi K3.1 Pro launch article | |
| 58%n = 6,000 | 42 | Artificial Analysis Independent evaluator · 2025-11-25 | Published by Artificial Analysis (2025-11-25).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Claude Opus 4.5 (thinking)' | |
| 64%n = 6,000 | 36 | Artificial Analysis Independent evaluator · 2025-11-17 | Published by Artificial Analysis (2025-11-17).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'Grok 4' | |
| 81%n = 6,000 | 19 | Artificial Analysis Independent evaluator · 2025-11-17 | Published by Artificial Analysis (2025-11-17).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience v1 (6,000 q); row 'GPT-5 (high)' | |
| 99.2%95% CI 98.5% to 99.8%n = 600 | 1 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationSAI run on the 600-question public subset (ArtificialAnalysis/AA-Omniscience-Public); grader google/gemini-2.5-flash with the paper's prompt; paper's answer system prompt; provider-default sampling; REASONING DISABLED (--no-reasoning) | |
| 26.3%n = 6,000 | read off a chart | Artificial Analysis (via BenchLM mirror) Independent evaluator · 2026-10-29 | Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'GLM-5.2' | |
| 25%n = 6,000 | read off a chart | Artificial Analysis (via BenchLM mirror) Independent evaluator · 2026-10-29 | Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'Grok 4.3' | |
| 27.1%n = 6,000 | read off a chart | Artificial Analysis (via BenchLM mirror) Independent evaluator · 2026-10-29 | Published by Artificial Analysis (via BenchLM mirror) (2026-10-29).Full configurationArtificial Analysis, 6,000 questions; AA-Omniscience Hallucination Rate (via BenchLM mirror); row 'Qwen3.8 Max Preview' |