CausalGame: Benchmarking Causal Thinking of LLM Agents in Games

Summary

CausalGame proposes a controlled benchmark for the "causal thinking" of LLM agents, cast as an interactive drone-design game whose survival dynamics are governed by an explicit structural causal model. The conceptual move is clean: existing AI-scientist benchmarks stress ideation, literature synthesis, coding, and open-ended discovery, but largely operate on observed variables in the absence of the three canonical obstacles — selection bias, latent confounding, measurement error — that motivated causal inference in the first place. By hand-designing 14 SCM-driven scenarios that make correlational strategies systematically misleading, and by tying both an outcome metric (Stage-2 survival on 1,000 drones) and an LLM-judged rubric to the same ground truth, the paper offers a well-motivated diagnostic instrument. The evaluation of 29 frontier models supports the headline claim, and a judge-independent configuration analysis on the Antenna Trap corroborates the failure through actual design-vector paths.

Executed replication strengthens the core story while sharpening two problems. On the positive side, an independent run reproduces the headline qualitatively (every reachable model sits well below threshold), reproduces the counter-intuitive antenna_def=0 mechanism almost exactly (82.3% vs 82.5%), and confirms the ~80% optimum is genuinely attainable for the antenna and DZ-categorical families — so the failure there is a model failure, not an unreachable ceiling. However, two families (Weather Noise and DZ Env-Shift) turn out to be unwinnable under the shipped SCM parameters: the best fixed design reaches only ~44% and ~64% against 55% and 70% thresholds, contradicting the "verified within ±2–3 pp" and threshold-margin claims and confounding "lack of causal thinking" with "unattainable target." The selection-bias sensitivity ordering (agent > prompt) also reverses under execution. Conceptually, the mode comparison bundles confounds, no classical causal-inference baseline is included, the CR rubric is scored by the very models the paper argues cannot reason causally, and much of the numeric infrastructure has internal inconsistencies. The contribution is real; the empirical claims need a careful, per-family pass.

Strengths

• Conceptual contribution. The paper isolates a genuine gap in AI-scientist evaluation: no existing benchmark systematically probes causal reasoning under selection bias, latent confounding, and measurement noise. Table 1 makes the comparison explicit, and Section 3.2 grounds each of the three challenges in a historical case (Berkson, Kipnis, Doll & Hill).
• SCM-first game design. The ground truth is an explicit per-scenario SCM with analytically derived optima, letting the win threshold be tied to a principled quantity. Execution confirms this works cleanly for the antenna and DZ-categorical families.
• Reproducible core mechanism. Independent re-runs recover the paper's signature counter-intuitive result — zeroing antenna defense to trigger "stealth mode" lifts survival from ~52% to ~82% — within sampling noise, which strongly validates the trap design and the diagnosis that agents chase the surface correlation.
• Dual outcome + rubric evaluation. Reporting both a hard survival metric and a rubric-based judgement lets the paper distinguish "won for the wrong reasons" from "won by identifying the mechanism," at least in principle, and the Pattern A–D taxonomy is a useful lens.
• Non-LLM calibration. Comparing to Default/Random/Uniform-High policies calibrates difficulty and yields the cutting finding that a constant policy beats several LLMs on the confounded DZ family — a finding the replication independently reproduces and, if anything, sharpens.
• Coverage. 29 models, two execution modes, an additional OpenCode framework, and per-cell means with CIs give the study broad empirical scope.

Weaknesses

• Two scenario families are effectively unwinnable as parameterized. Executed replication shows the Weather Noise optimum tops out at ~44–46% (win threshold 55%) and DZ Env-Shift at ~64% (threshold 70%) under the shipped SCM, contradicting the "verified within ±2–3 pp" claim (Section 3.1/Appendix C.4) and the "~23 pp" weather margin (Figure 23). On these families, "models fail because they lack causal thinking" is confounded with "no design can win," and Weather Noise's documented optimum is a deployment policy, not a design — so it may test noise-averaging rather than causal-structure recovery.
• A key supporting claim reverses under execution. The "agent mode is more sensitive to selection bias" claim (with its compounding-feedback hypothesis) is contradicted: the replicated selection-bias delta is smaller for agent mode than for prompt mode, and the effect was never paired-tested.
• Mode comparison is a confound bundle. Table 5 lists five simultaneous differences between Agentic and Prompting modes, yet Observation 3 draws a per-model conclusion about "agentic framework" benefit. This should be reframed as descriptive or replaced with a single-dimension ablation.
• No classical causal-inference baseline. LLMs are compared only to random/constant/random-then-LLM policies. A scripted do-calculus/PC policy on the same budget is the natural upper bound needed to separate "LLMs lack causal thinking" from "the task is hard for anyone."
• LLM-judge circularity and rubric binarization. The 11-of-16-point Causal Reasoning rubric is scored by LLMs the paper argues cannot reason causally, with only moderate CR inter-rater agreement (0.61–0.64) versus 0.85–0.89 for mechanical items; and the CR items collapse partial understanding and no-engagement into the same zero (87–92% zeros).
• Statistical overclaims. "Significantly below the win threshold" is asserted without a formal test; the reasoning-effort ordering claims rest on few-point deltas at n = 3 (CI half-widths ~12–37 pp); and Figure 8's Spearman correlations carry no CIs or p-values despite sign-based conclusions.
• Definitional gaps. The SCM formalism sentence in Section 3.1 ends mid-sentence, and the selection-bias definition is self-referential; the "all core variables observable" statement (Appendix C.5.4) appears to contradict the hidden-confounder/measurement-error design.
• Internal numerical inconsistencies. CI half-widths differing by a factor of ~2.5 between two tables; Prompt-style failure-mode percentages summing to 97.1%; antenna configuration buckets summing to 82%; win counts not summing to 14; tool names and tool-iteration budget disagreeing across sections; difficulty labels contradicting the numeric ranking; and failed-trajectory transcripts using invalid design keys.

Reproducibility & code

• Headline aggregates (confirmed qualitatively). On the 6 of 29 models reachable given available API keys (2 modes × 5 scenarios × 3 trials), every model×mode cell mean fell in ~53–64% with no cell reaching 75%, reproducing the central failure claim; the universal quantifier over 29 models remains an inference from a ~21% sample.
• Optimum attainability (mixed). Deploying the documented optimal designs reproduced antenna_trap (82.5%) and DZ-categorical (81.1%) as wins, but Weather Noise measured ~44.2% (ceiling ~46.3%) against a 55% threshold and DZ Env-Shift ~64.1% against 70% — both unwinnable under the shipped parameters, undercutting the "±2–3 pp verified" and threshold-margin claims for those families.
• Example trajectories (confirmed). The antenna_def=0 vs antenna_def=10 contrast reproduced at 82.3% vs 52.1% (paper 82.5% vs 53.2%), within sampling noise.
• Selection-bias contrast (partial / reversed). The Prompt column reproduced closely (+8.6 vs +7.8 pp) but the Agent column came in lower (+7.4 vs +9.5 pp), and the "agent more sensitive" ordering reversed.
• Non-LLM baselines (partial). Default reproduced (49.8% vs 49.0%); Random (45.1%) and Uniform-High (59.3%) deviated, with Uniform-High clearing 75% on 5/6 DZ-categorical scenarios — a stronger version of the paper's own finding, but a deviation from the reported 52.7%.
• Missing pipelines. The released code does not ship the LLM-judge/rubric scorer, the ICC computation, the failure-mode classifier (Patterns A–D, Figures 10–11), the cross-benchmark correlation code (Figure 8), the non-LLM baseline scripts, or per-trial outputs/aggregation code — so Section 4.4, Figure 8, and the exact aggregates in Tables 3 and 12–17 could not be verified. The OpenCode +6.9 pp gain was not exercised.
• Agent-mode censoring. 45 of 90 hybrid runs never submitted a design (vs 2/90 in prompting), so reported agent-mode means are conditioned on submission — a survivorship bias in the numbers themselves. No defects were found in the paper's own SCM/evaluation code; the replicator's own harness bugs were corrected before these numbers were produced.

Recommended Changes

Essential

• Re-derive and re-verify per-family optima under the shipped SCM. For Weather Noise and DZ Env-Shift, either supply an actual optimal design that clears the threshold or lower/relabel the thresholds, and correct the "±2–3 pp verified" and Figure 23 margin claims. Where no design can win, state that explicitly so those families are not read as pure causal-reasoning failures. (Addresses the unwinnable-families weakness and the optimum/threshold reproducibility bullets.)
• Drop or re-derive the "agent mode is more sensitive to selection bias" claim with a per-model paired difference-in-differences and CI, since execution reversed the ordering. (Addresses the reversed-claim weakness.)
• Add a classical causal-inference baseline (scripted do-calculus/PC on the 200-drone budget) to isolate "LLMs lack causal thinking" from "task is hard for any procedure." (Addresses the missing-baseline weakness.)
• Validate the LLM judges against a hand-labeled gold set (or a symbolic SCM checker) and report agreement, at least for the CR items. (Addresses the judge-circularity weakness.)
• Release the missing code and data: the judge/rubric scorer, ICC code, the rubric-to-Pattern classifier, the Figure 8 correlation script and external scores, the non-LLM baseline scripts, and per-trial CSVs with the aggregation code for Tables 3 and 12–17. (Addresses the "missing pipelines" and "headline aggregates" reproducibility bullets.)
• Report and account for agent-mode no-submits. Give per-cell no-submit rates and state how no-submits enter averages and win counts, since half of agentic runs never submitted. (Addresses the agent-mode censoring bullet.)
• Replace "significantly below threshold" with a real test (one-sided per-model test vs threshold) or drop the significance wording. (Addresses the statistical-overclaim weakness.)
• Reframe Observation 3 as descriptive or add a single-dimension controlled ablation. (Addresses the mode-confound weakness.)

Suggested

• Add graduated CR credit so recognizing a trap without a full chain is not scored zero, then re-report the CR distribution. (Addresses rubric binarization.)
• Add bootstrapped CIs to Figure 8 and soften sign-based interpretations. (Addresses correlation significance.)
• Present within-family ordering claims as suggestive or increase trials for the reasoning-effort comparisons. (Addresses n = 3 fragility.)
• Fix the CI-table mislabel (68.6 ± 5.9 vs ± 14.7) to a single, well-labeled quantity. (Addresses the CI-inconsistency weakness.)
• Complete the SCM definition sentence, rewrite the selection-bias definition, and clarify "core variables" so it does not contradict the hidden-confounder design. (Addresses the definitional-gaps weakness.)
• Document the OpenCode configuration (version, tools, prompt, budget) and report its gain per family. (Addresses the OpenCode reproducibility bullet.)
• Reconcile the small internal inconsistencies: Pattern D fraction for Prompt style, the 82% configuration buckets, the win-count/tie totals, tool names and tool-iteration budget, Table 17 difficulty bins, and the invalid design keys in the failed-trajectory transcript. (Addresses the internal-inconsistency weakness.)