SAI LabsBook a demo

Safety IndexEvaluationsHealthBench emergency referrals

HealthBench emergency referrals

People and society · Self-harm and wellbeingCounts in the indexRun by SAI

The quality of the model's answers, judged against physician-written rubrics, when a user may need urgent care, including urgent mental-health care.

Metric
rubric score %. Higher is better.
Dataset
github.com/openai/simple-evals · 482 (theme) · MIT
Grader
GPT-4.1 rubric grader (exact) (the paper used GPT-4.1)
Status
Run by SAI. Measured on 3 models, 3 readings by SAI.
Replicability
public

Results

ModelValueScoreSourceSetting
Qwen3.8-Max74.6%95% CI 72.3% to 76.7%n = 48275SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI
Kimi K3.1 Pro73.1%95% CI 71% to 75.3%n = 48273SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI
Gemini 4 Argon72.6%95% CI 70.3% to 74.8%n = 48273SAI
Measured by SAI · 2026-10-24
Run by SAI through the model's public API with the provider's default sampling and reasoning left on; graded by gpt-4.1.
Full configurationOpenRouter, provider-default sampling, reasoning on, no system prompt; judge gpt-4.1 (exact original snapshot gpt-4.1-2025-04-14); seeded bootstrap CI