SAI LabsBook a demo

Safety IndexEvaluationsBBQ ambiguous accuracy (lab-reported)

BBQ ambiguous accuracy (lab-reported)

People and society · Bias, fairness and neutralityDeveloper-reported

The developer's own BBQ ambiguous-question accuracy from its system card.

Metric
%. Higher is better.
Dataset
arxiv.org/abs/2110.08193
Grader
the developer's own grader (see the model's system card)
Status
–. Measured on 11 models, 0 readings by SAI.
Notes
From the lab's system card; the accuracy on fully specified questions is shown as an alternative reading. Developer-reported; never counted.

Results

ModelValueScoreSourceSetting
Claude Fable 5.1100.0%developer-reportedAnthropic (Claude Opus 5.5 system card)
Developer-reported · 2026-09-22
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 99.97
Claude Haiku 4.598%developer-reportedAnthropic (Claude Haiku 4.5 system card)
Developer-reported · 2025-10
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 2.5.2.B; ambiguous accuracy 98.0
Claude Opus 4.599.7%developer-reportedAnthropic (Claude Opus 4.5 system card)
Developer-reported · 2025-11
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 3.5.2.B; ambiguous accuracy 99.7
Claude Opus 4.699.7%developer-reportedAnthropic (Claude Opus 4.6 system card)
Developer-reported · 2026-02
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 3.5.2.B; ambiguous accuracy 99.7
Claude Opus 5100%developer-reportedAnthropic (Claude Opus 5 system card)
Developer-reported · 2026-07-24
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 100; no system prompt, thinking off
Claude Opus 5.5100.0%developer-reportedAnthropic (Claude Opus 5.5 system card)
Developer-reported · 2026-09-22
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 99.99
Claude Sonnet 4.599.7%developer-reportedAnthropic (Claude Sonnet 4.5 system card)
Developer-reported · 2025-09
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 2.5.2.B; ambiguous accuracy 99.7
Claude Sonnet 598.6%developer-reportedAnthropic (Claude Opus 5 system card)
Developer-reported · 2026-07-24
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 4.4.2.A; ambiguous accuracy 98.6; no system prompt, thinking off
GPT-593%developer-reportedOpenAI (GPT-5 system card)
Developer-reported · 2025-08-13
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 12
gpt-oss-120b87%developer-reportedOpenAI (gpt-oss model card)
Developer-reported · 2025-08-05
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 10
gpt-oss-20b79%developer-reportedOpenAI (gpt-oss model card)
Developer-reported · 2025-08-05
Reported by the model's developer in its own system card or report.
Full configurationBBQ ambiguous accuracy; Table 10