SAI LabsBook a demo

Safety IndexEvaluationsSimpleQA Verified

SimpleQA Verified

Interaction and honesty · Factuality and hallucinationCounts in the indexRun by SAI

How often the model gives a wrong answer to 1,000 short factual questions, rather than answering correctly or saying it does not know.

Metric
% incorrect. Lower is better.
Dataset
www.kaggle.com/benchmarks/deepmind/simpleqa-verified · 1,000 · MIT (HF mirror)
Grader
GPT-4.1 (benchmark's own grader) (the paper used gpt-4.1-2025-04-14)
Status
Run by SAI. Measured on 4 models, 4 readings by SAI.
Replicability
public

Results

ModelValueScoreSourceSetting
Kimi K315.4%95% CI 13.2% to 17.7%n = 1,00085SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1
Gemini 4 Argon15.5%95% CI 13.3% to 17.8%n = 1,00085SAI
Measured by SAI · 2026-10-23
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge gpt-4.1 (= reference gpt-4.1-2025-04-14)
Qwen3.8-Max52.2%95% CI 49.3% to 55%n = 1,00048SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1
Qwen3.5-9B90.4%95% CI 88.5% to 92.3%n = 1,00010SAI
Measured by SAI · 2026-10-22
Run by SAI through the model's public API with the provider's default sampling and reasoning left on.
Full configurationofficial SimpleQA Verified autorater prompt; 1,000 questions; no tools, no system prompt; provider-default sampling; judge openai/gpt-4.1; REASONING DISABLED (--no-reasoning) via OpenRouter, not the local engine