BBQ
People and society · Bias, fairness and neutralityRetired from the mean (saturated)Run by SAI
Whether the model answers 'unknown' on under-specified questions instead of guessing a stereotype.
- Metric
- % accuracy (ambiguous). Higher is better.
- Dataset
- github.com/nyu-mll/BBQ · 1,000 sample (58,492 full) · CC-BY-4.0
- Grader
- exact match (the paper used same)
- Status
- Run by SAI. Measured on 14 models, 3 readings by SAI.
- Replicability
- public
- Notes
- Two readings: SAI engine runs on the full question set and Stanford HELM Safety on a 1,000-question sample.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 97.8% | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; SAI engine, the public evaluation code bbq full set (~58k items); judge none | |
| 97.2% | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; SAI engine, the public evaluation code bbq full set (~58k items); judge none | |
| 96.3% | not scored | SAI Measured by SAI · 2026-10-23 | Run by SAI on the open-weight model served locally; graded by none.Full configurationvLLM bf16, one 80 GB GPU, thinking on, no system prompt, the public evaluation code default temperatures; full set; SAI engine, the public evaluation code bbq full set (~58k items); judge none | |
| 92.8%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Haiku (20251001)' | |
| 98.9%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Claude 4.5 Sonnet (20250929)' | |
| 96.6%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'DeepSeek R1' | |
| 98.4%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Gemini 3 Pro (Preview)' | |
| 96.8%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'GPT-5 (2025-08-07)' | |
| 88.7%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'GPT-5.1 (2025-11-13)' | |
| 98.5%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-120b' | |
| 96.7%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'gpt-oss-20b' | |
| 93.7%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Grok 4 (0709)' | |
| 93%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Llama 4 Maverick (17Bx128E) Instruct FP8' | |
| 96.6%n = 1,000 | not scored | Stanford CRFM HELM Safety v1.17.0 Independent evaluator · 2025-11-24 | Published by Stanford CRFM HELM Safety v1.17.0 (2025-11-24).Full configurationHELM Safety v1.17.0; HELM row 'Qwen3 235B A22B FP8 Throughput' |