Safety IndexEvaluationsSimpleQA Verified
SimpleQA Verified
Interaction and honesty · Factuality and hallucinationCounts in the indexRun by SAI
How often the model gives a wrong answer to 1,000 short factual questions, rather than answering correctly or saying it does not know.
- Metric
- % incorrect. Lower is better.
- Dataset
- www.kaggle.com/benchmarks/deepmind/simpleqa-verified · 1,000 · MIT (HF mirror)
- Grader
- GPT-4.1 (benchmark's own grader) (the paper used gpt-4.1-2025-04-14)
- Status
- Run by SAI. Measured on 4 models, 4 readings by SAI.
- Replicability
- public
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 15.4%95% CI 13.2% to 17.7%n = 1,000 | 85 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1 | |
| 15.5%95% CI 13.3% to 17.8%n = 1,000 | 85 | SAI Measured by SAI · 2026-10-23 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge gpt-4.1 (= reference gpt-4.1-2025-04-14) | |
| 52.2%95% CI 49.3% to 55%n = 1,000 | 48 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1 | |
| 90.4%95% CI 88.5% to 92.3%n = 1,000 | 10 | SAI Measured by SAI · 2026-10-22 | Run by SAI through the model's public API with the provider's default sampling and reasoning left on.Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1; REASONING DISABLED (--no-reasoning) via OpenRouter, not the local engine |