SAI LabsBook a demo

Safety IndexEvaluationsBBQ

BBQ

People and society · Bias, fairness and neutralityRetired from the mean (saturated)Run by SAI

Whether the model answers 'unknown' on under-specified questions instead of guessing a stereotype.

Metric
% accuracy (ambiguous). Higher is better.
Dataset
github.com/nyu-mll/BBQ · 1,000 sample (58,492 full) · CC-BY-4.0
Grader
exact match (the paper used same)
Status
Run by SAI. Measured on 14 models, 3 readings by SAI.
Replicability
public
Notes
Two readings: SAI engine runs on the full question set and Stanford HELM Safety on a 1,000-question sample.

Results

ModelValueScoreSourceSetting
Qwen3.5-4B97.8%not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; SAI engine, the public evaluation code bbq full set (~58k items); judge none
Qwen3.5-9B97.2%not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; SAI engine, the public evaluation code bbq full set (~58k items); judge none
Qwen3.8-27B96.3%not scoredSAI
Measured by SAI · 2026-10-23
Run by SAI on the open-weight model served locally; graded by none.
Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; SAI engine, the public evaluation code bbq full set (~58k items); judge none
Claude Haiku 4.592.8%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Haiku (20251001)'
Claude Sonnet 4.598.9%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Sonnet (20250929)'
DeepSeek R196.6%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1'
Gemini 3 Pro98.4%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Gemini 3 Pro (Preview)'
GPT-596.8%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'GPT-5 (2025-08-07)'
GPT-5.188.7%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'GPT-5.1 (2025-11-13)'
gpt-oss-120b98.5%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-120b'
gpt-oss-20b96.7%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-20b'
Grok 493.7%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)'
Llama 4 Maverick93%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Llama 4 Maverick (17Bx128E) Instruct FP8'
Qwen3-235B-A22B96.6%n = 1,000not scoredStanford CRFM HELM Safety v1.17.0
Independent evaluator · 2025-11-24
Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).
Full configurationHELM Safety v1.17.0; HELM row 'Qwen3 235B A22B FP8 Throughput'