SAI LabsBook a demo

Safety IndexEvaluationsTruthfulQA MC1

TruthfulQA MC1

Interaction and honesty · Factuality and hallucinationRetired from the mean (saturated)Run by SAI

How often the model picks the true answer rather than a common misconception on 817 trick questions.

Metric
% accuracy. Higher is better.
Dataset
github.com/sylinrl/TruthfulQA · 817 · Apache-2.0
Grader
deterministic MC
Status
Run by SAI. Measured on 4 models, 5 readings by SAI.
Replicability
public
Notes
Saturated and widely trained on; shown for continuity only.

Results

ModelValueScoreSourceSetting
Qwen3-8B70.9%n = 817not scoredSAI
Measured by SAI · 2026-10-20
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM, thinking on unless noted, no system prompt; research run; judge none
Qwen3-8B71.4%n = 817not scoredSAI
Measured by SAI · 2026-10-21
Run by SAI on the open-weight model served locally; graded by none.
Full configurationSAI engine, local vLLM serving, thinking on, no system prompt, default sampling; full set; judge none (multiple choice); stderr 1.58 points
Qwen3.5-4B74.7%n = 817not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none
Qwen3.5-9B80.4%n = 817not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none
Qwen3.8-27B79.9%n = 817not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; judge none