Safety IndexEvaluationsBBQ ambiguous accuracy (lab-reported)
BBQ ambiguous accuracy (lab-reported)
People and society · Bias, fairness and neutralityDeveloper-reported
The developer's own BBQ ambiguous-question accuracy from its system card.
- Metric
- %. Higher is better.
- Dataset
- arxiv.org/abs/2110.08193
- Grader
- the developer's own grader (see the model's system card)
- Status
- –. Measured on 11 models, 0 readings by SAI.
- Notes
- From the lab's system card; the accuracy on fully specified questions is shown as an alternative reading. Developer-reported; never counted.
Results
| Model | Value | Score | Source | Setting |
|---|---|---|---|---|
| 100.0% | developer-reported | Anthropic (Claude Opus 5.5 system card) Developer-reported · 2026-09-22 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 99.97 | |
| 98% | developer-reported | Anthropic (Claude Haiku 4.5 system card) Developer-reported · 2025-10 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 2.5.2.B; ambiguous accuracy 98.0 | |
| 99.7% | developer-reported | Anthropic (Claude Opus 4.5 system card) Developer-reported · 2025-11 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 3.5.2.B; ambiguous accuracy 99.7 | |
| 99.7% | developer-reported | Anthropic (Claude Opus 4.6 system card) Developer-reported · 2026-02 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 3.5.2.B; ambiguous accuracy 99.7 | |
| 100% | developer-reported | Anthropic (Claude Opus 5 system card) Developer-reported · 2026-07-24 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 100; no system prompt, thinking off | |
| 100.0% | developer-reported | Anthropic (Claude Opus 5.5 system card) Developer-reported · 2026-09-22 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 99.99 | |
| 99.7% | developer-reported | Anthropic (Claude Sonnet 4.5 system card) Developer-reported · 2025-09 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 2.5.2.B; ambiguous accuracy 99.7 | |
| 98.6% | developer-reported | Anthropic (Claude Opus 5 system card) Developer-reported · 2026-07-24 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 98.6; no system prompt, thinking off | |
| 93% | developer-reported | OpenAI (GPT-5 system card) Developer-reported · 2025-08-13 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 12 | |
| 87% | developer-reported | OpenAI (gpt-oss model card) Developer-reported · 2025-08-05 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 10 | |
| 79% | developer-reported | OpenAI (gpt-oss model card) Developer-reported · 2025-08-05 | Reported by the model's developer in its own system card or report.Full configurationBBQ ambiguous accuracy; Table 10 |