Large Language Models Develop Novel Social Biases Through Adaptive Exploration

Summary

The paper argues that debiasing LLMs by scrubbing existing stereotypes is not sufficient: as models are placed into stateful, sequential decision loops, they will invent new biases from noisy feedback in the same way humans do in the iterative-hiring paradigm of Bai et al. (2022a; 2025a). The authors instantiate this claim on ~13 frontier and predecessor models across three complementary metrics (Stratification Index, Between-Group Divergence, Group Assignment Stochasticity Index), and find that (i) frontier models stratify equal, artificial demographic groups into job quadrants substantially more than humans and the standard Thompson / UCB1 baselines; (ii) within each of six model families, newer and larger models stratify more; (iii) the pattern is orthogonal to the BBQ single-prompt bias benchmark; and (iv) among a slate of interventions, only an explicit diversity-bonus term in the objective robustly recovers near-random allocations, while temperature, CoT, and context compression do not. The move to reframe LLM social bias as an allocational phenomenon that grows over stateful interactions — rather than a representational leak from pretraining — is the paper's main conceptual contribution, and it is well-motivated relative to the current benchmark-first fairness literature. The results are largely consistent with recent bandit/agentic work on greedy LLM commitment (Krishnamurthy et al. 2024; Schmied et al. 2026), which is both the strength and the main conceptual limitation: it is not fully argued why the labeled-hiring stratification is a distinct social-bias phenomenon rather than the same greedy-exploitation failure dressed in demographic language, especially given that Section 5.2's abstract A-D / C1-C4 variant collapses to one-arm commitment. Methodologically, the study is broad but light on statistical detail (tests unnamed, effect sizes absent, some 'significance' claims stand on two point estimates without CIs), and the reproducibility posture is thin: no code, parsers, trajectories, or API snapshot dates are released with the manuscript. The contribution is valuable and the diversity-bonus finding is genuinely striking; addressing the framing and statistical gaps below would materially strengthen it.

Strengths

• Conceptual reframing. The distinction between representational bias (leaking from pretraining) and allocational bias formed under sequential decision-making is a clean, non-obvious way to describe a phenomenon that current bias benchmarks miss, and connects the LLM literature to classical work on statistical discrimination and feedback loops (Phelps 1972; O'Neil 2016).
• Metric design. SI, BGD, and GASI are complementary and (mostly) well-formalized: SI reduces to expected mutual information under uniform job marginals (Appendix B.1.1), GASI cleanly separates within-run emergence from prior associations, and the calibration curves in Figures 8-10 give tangible readings for their ranges.
• Breadth of models and interventions. The evaluation spans 6 model families, direct/CoT/reasoning prompts, three cover-story domains (hiring, refugee resettlement, military conscription), an abstract-bandit variant, an unequal-success-rate variant, temperature/T=0/CoT/context-length/richer-features/prompt-steer ablations, and a real-demographic sanity check. Few bias papers touch this many axes.
• Clean identifying evidence for emergence. Appendices G, H, and the GASI comparison collectively rule out prior demographic associations, sampling variance, and job-trajectory variance as the drivers of stratification, isolating the binary success/failure signal as the actual source. This is the cleanest kind of mechanism argument for the paper's central claim.
• Diversity-bonus intervention. Beyond identifying failure, the paper offers a concrete, deployable intervention that dominates value-steering, fairness prompts, and shared-norm prompts on SI, while being trivial to implement. Even given the Appendix K caveat, this is a useful practical result.

Weaknesses

• Greedy-commitment vs. novel-bias framing. Section 5.2's abstract A-D / C1-C4 setup shows models collapse into a single-arm strategy without any demographic labels, which is exactly the greedy failure mode Krishnamurthy et al. (2024) and Schmied et al. (2026) already document. The paper never argues why the labeled-hiring stratification is a different phenomenon rather than a re-labeled instance of the same greedy contextual-bandit failure. This substantially affects how strong the 'novel social bias' framing lands.
• Reasoning-model claim is fragile. The 'newer/larger/reasoning ⇒ more stratified' argument, echoed in the abstract, rests on only two reasoning models (o3, DeepSeek-R1) reported as point estimates (1.83 and 1.41) without 95% CIs, unlike every other headline SI in Section 4.1. Given how central this is to the discussion, either matched-protocol CIs or at least one additional reasoning model per family is needed.
• BBQ correlation is presented but not quantified. Figure 4's negative-SI-vs-BBQ picture drives an important 'capability improves standard benchmarks but worsens emergent bias' claim, yet no per-model BBQ scores are tabulated, no source is cited for the BBQ scores of GPT-5.4 / Claude 4.6 Sonnet / Gemini 3 Flash (which do not appear in Parrish et al. 2022 or Liang et al. 2023), and no correlation coefficient or significance test is reported.
• 'Richer information' variant is not actually informative. In the age/education refugee variant, features are sampled randomly and independently from success outcomes, so they carry zero mutual information with the reward. Any SI reduction cannot then reflect the model 'explaining observations via other features' — it can only reflect displacement of stratification onto distractor features. The current framing overstates what the intervention shows.
• Diversity-steer trade-off is understated in the main text. Appendix K demonstrates that the recommended diversity bonus meaningfully hurts success rates when a genuine demographic-job mapping exists. Section 5.3 advertises the SI drop without quantifying this cost, even though the trade-off is the operationally important part for anyone deploying the intervention.
• Statistical procedure under-specified. The paper reports 'not statistically significant' or 'p = 0.04' or 'statistically significantly more' in many places without naming the test (t-test / bootstrap / permutation), stating what family Benjamini-Hochberg is applied over, or reporting effect sizes. For the 80-round vs. 40-round comparison in Appendix I this is particularly damaging, because the null claim ('doubling the horizon doesn't help') is load-bearing.
• UCB1 baseline sits below Random. Table 3's UCB1 rows are all below the Random SI, which is not what a naive reader expects from an 'exploration algorithm'. The paper reports UCB1 as 'substantially lower' than Thompson/humans/LLMs but omits this fact, which matters because it means UCB1 is not obviously the right rational benchmark to hold LLMs against.
• Reporting details for small tables. Table 4 calls a 0.30→0.47 GASI jump 'comparable', Table 10 spans a 0.95–4.9 range that the prose describes as '1 to 5', and Table 7's Claude Direct = 0.234 is called 'well below the random baseline' (0.25). These are individually minor but compound to a sense that the numbers are being rounded to the narrative.
• Overclaimed generalization. 'Not attenuated by agentic scaffolding such as ReAct, but instead persists across reasoning paradigms' is asserted from two models with no matched non-agentic baseline quoted in-line. Similar single- or two-model claims (unequal-success-rate accuracy, elicited-prior success probabilities) are stated as if they apply to LLMs generally.
• Human baseline is inherited. The human SI/BGD/GASI come from Bai et al. (2025a); no new participants were collected. The 200-vs-30 sample-size asymmetry between humans and LLMs, and the platform (survey vs. chat) difference, are not acknowledged when the paper repeatedly claims LLMs stratify 'more than humans'.
• Impact statement. For a paper whose core finding is that LLMs generate stratified allocations under deployment-plausible conditions and whose recommended intervention has a real failure mode, 'no envisioned negative ethical implications' is under-considered.

Reproducibility & code

The mode here is code read, not executed. No code repository or trajectory dump is shipped with the manuscript; the released-artifact directory is empty, so all reproducibility assessment reduces to what can be read off the manuscript itself.

• No shipped code, parsers, or trajectories. ~13 model configurations, three prompt families, and dozens of ablation conditions are described in prose and appendix prompts, but the harness that issued the API calls, the parser that maps free-text LLM output back to {Tufa, Aima, Reku, Weki} × {job}, and the per-run allocation matrices that produce every SI/BGD/GASI number are unavailable. Even the bandit baselines (Thompson, UCB1) are documented only by algorithm names.
• API snapshot identifiers missing. Models are referred to only by marketing name ('GPT-5.4', 'Claude Sonnet 4.6', 'Gemini 3 Flash', 'OpenAI o3', 'DeepSeek-R1'); vendors reroute these names to new snapshots without notice, and 'default medium reasoning effort' means different things across OpenAI and DeepSeek. For Qwen 2.5-72B and Llama 4 Maverick the hosting stack (Together AI 'Instruct Turbo' vs. local weights, quantization) is not stated, though it materially affects sampling.
• Statistical procedure gaps. The Benjamini-Hochberg footnote does not specify (a) the base test used for each comparison, (b) the family over which BH is applied, or (c) how missing data (e.g., Claude refusing the system-prompt steer, CoT 'gibberish' at T=1.5) enters the aggregate.
• Appendix D matrices are undocumented. The rank-ordered allocation tables that carry the 'newer ⇒ more identity-like' visual argument lack a legend for row/column semantics, no note on whether counts are attempts or successes, and inconsistent layout across families.
• Appendix A.7 typo. Two consecutive 'Successful Outcome Prompt' headers describe successful and unsuccessful placements respectively; the second is presumably 'Unsuccessful Outcome Prompt.' This is the sole source of the conscription outcome text, so reproducers may copy the wrong label.

None of these are outright fatal to reproduction — the prompts, models, run counts, and metric formulas are largely specified — but together they push the study's re-execution cost from 'straightforward' to 'expensive and version-fragile', and prevent any independent verification of the SI/BGD/GASI numbers as reported.

Recommended Changes

Essential

• Address the greedy-commitment alternative. In Section 4 or 6, argue explicitly why labeled-hiring stratification is a different phenomenon from the greedy contextual-bandit failure documented by Krishnamurthy et al. (2024) and Schmied et al. (2026) — ideally by comparing SI in the labeled setting against SI in a matched unlabeled/relabeled contextual bandit, so the reader can see whether the demographic labels add signal beyond generic greediness.
• Report CIs and add at least one reasoning model per family. For OpenAI o3, DeepSeek-R1 (and any new reasoning entries), give SI/BGD with the same 30-run 95% CI protocol used for non-reasoning models, and add at least one other reasoning configuration per family so the 'reasoning ⇒ more stratified' claim rests on more than two point estimates.
• Tabulate BBQ scores in Figure 4 and quantify the correlation. Add a table with per-model BBQ score, source, and evaluation protocol, and report a rank correlation (with p-value) between BBQ and SI, so the 'BBQ up, SI up' visual becomes a testable statistic.
• Fix the 'richer information' framing. Either sample age/education from an outcome-informative distribution in the refugee variant, or reframe the finding as 'adding surface distractor features displaces stratification' rather than 'LLMs use available features to explain outcomes.'
• Quantify the Appendix K trade-off in Section 5.3. In the main-text description of the diversity-bonus steer, report the success-rate cost from Figure 12 alongside the SI reduction, so practitioners see both sides.
• State every statistical test. For each 'significant / not significant' claim, name the test, the family over which BH is applied, and (ideally) the effect size. Special priority: the 80-round vs. 40-round null in Appendix I, the CoT vs. Direct comparisons in Figure 2, and the Table 2 CI-overlap bolding rule.
• Release code and per-run trajectories. Ship a repository with the query harness, response parser, SI/BGD/GASI implementation, per-run allocation matrices, and per-model API snapshot/date/provider identifiers, so that the numbers in Figures 2-6 and Tables 1-11 can be independently checked.

Suggested

• Add API snapshot table. Replace the marketing-name model list in Section 3.3 with a table giving the API date, endpoint identifier, provider, quantization / hosting stack, and reasoning-effort setting for each model.
• Anchor SI to its theoretical maximum. Note in Section 3.2 that with 4 groups and 4 quadrants max SI = 2 bits, and consider reporting normalized SI alongside raw SI so '1.83' immediately reads as '92% of the way to complete lock-in'.
• Note the UCB1-vs-Random ordering. Add one sentence in Appendix E explaining that UCB1 sits below Random because it enforces at-least-one-pull-per-arm exploration, so the reader is not misled into treating UCB as a stronger 'rational' benchmark than Random.
• Clarify the diversity-bonus formula. State exactly what 'sector of jobs' means in Appendix A.4.1's denominator (specific job title vs. warmth-competence quadrant vs. something else), so the incentive geometry is unambiguous.
• Restore the Lemma 1 display equations. Include the missing SI definition and the intermediate equations in the proof body so the derivation is checkable.
• Fix Table 10 framing. Either drop GPT-4o CoT (0.95) from the '1 to 5 consecutive failures' range or state the observed range as '~1 to 5' with the sub-1 case noted; report the probability at the mean (2.6), not just at the extreme (5), for the 10^{-5} claim.
• Soften Appendix H language. Report Table 7's Claude Direct (0.234) as 'comparable to' rather than 'well below' the Random baseline (0.25), and flag it as the closest cell to the random threshold.
• Handle refusals in Figure 6. Annotate the Claude 4 Sonnet missing cell in Figure 6 and say in the caption how it is imputed / dropped.
• Quantify 'gibberish' at T=1.5. Report the parse-failure rate for CoT at T=1.5 / 1.2 and, if any responses parse cleanly, report SI on that subset.
• Legend for Appendix D matrices. Add a single legend describing row/column semantics and what the integers count, and unify layout across families; ideally verify that row sums equal 40 per run.
• Acknowledge the inherited human baseline. Add a sentence in Section 3.1 or Appendix C noting the sample-size asymmetry (200 humans/condition vs. 30 LLM runs) and the platform difference between the survey and the chat interface, when comparing LLMs to humans on SI/BGD/GASI.
• Rewrite the Impact Statement. Engage seriously with foreseeable misuse: misapplication of the diversity steer where real group differences exist (Appendix K), and downstream uses of the stereotype categories introduced for Appendix M.
• Fix the '20 jobs / 40 rounds' phrasing. Reword the opening of Section 3.1 so the reader does not misread 'twenty jobs' as the horizon length.
• Fix the second '## Successful Outcome Prompt' header in Appendix A.7 to read 'Unsuccessful Outcome Prompt'.