Safety IndexEvaluationsTruthfulQA MC1
TruthfulQA MC1
Interaction and honesty · Factuality and hallucinationRetired from the mean (saturated)Run by SAI
How often the model picks the true answer rather than a common misconception on 817 trick questions.
- Metric
- % accuracy. Higher is better.
- Dataset
- github.com/sylinrl/TruthfulQA · 817 · Apache-2.0
- Grader
- deterministic MC
- Status
- Run by SAI. Measured on 4 models, 5 readings by SAI.
- Replicability
- public
- Notes
- Saturated and widely trained on; shown for continuity only.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 70.9%n = 817 | not scored | SAI Measured by SAI · 2026-10-20 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM, thinking on unless noted, no system prompt; research run; judge none | |
| 71.4%n = 817 | not scored | SAI Measured by SAI · 2026-10-21 | Run by SAI on the open-weight model served locally; graded by none.Full configurationSAI engine, local vLLM serving, thinking on, no system prompt, default sampling; full set; judge none (multiple choice); stderr 1.58 points | |
| 74.7%n = 817 | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none | |
| 80.4%n = 817 | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none | |
| 79.9%n = 817 | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none |