DISCO: Mitigating Bias in Deep Learning with Conditional Distance Correlation

Summary

The paper makes two contributions. First, it introduces the Standard Anti-Causal Model (SAM), a graphical framework in which the target $Y$ generates $X$ and biases split into non-directed backdoor variables $\mathbf{Z}$ and mediators $\mathbf{W}$. Working within SAM, the authors decompose the total variation of predictions into counterfactually stable, indirect, and spurious components, and show that under standard assumptions (no unobserved backdoors, no valid proxies in $X$) the observational criterion $\hat{Y} \perp \mathbf{B} \mid Y$ is sufficient to zero out the two spurious components — a slightly stronger conclusion than earlier necessity results (Veitch et al., 2021) and a structural refinement of Makar & D'Amour's separation criterion. Second, the paper closes the practical gap that had made conditional distance correlation (Wang et al., 2015) unusable inside gradient-based optimization by proposing two GPU-friendly estimators: DISCO$_m$, which averages local V-statistics over an $m$-sample reference subset, and sDISCO, which claims an exact algebraic factorization that keeps memory at $O(n^2)$. Empirically the authors benchmark against seven baselines (GDRO, IRM, Fishr, c-MMD, HSCIC, CIRCE, an adversarial head) across six datasets spanning regression, classification, vision and NLP, and add a controlled pathway-counterfactual analysis on dSprites.

The conceptual move — treating conditional independence with respect to $Y$ as the pathway-blocking criterion, and then engineering a batch-scale estimator for it — is genuinely useful, and independent execution of the estimator on the controlled dSprites setup confirms the qualitative causal-stability story: blocking both known biases nearly zeros counterfactual sensitivity while blocking one leaves residual on the unobserved path. That said, the manuscript as delivered has substantive presentation and content problems. Multiple central display equations are missing (path-specific effect definitions, the sDISCO factorization, the counterfactual metrics); the notation used in Sections 4.4/H.4 collides with the paper's own; the sDISCO grand-mean identity does not verify at index level; the consistency proof conflates the continuous mapping theorem with Slutsky's and treats strongly dependent local estimators as i.i.d.; the corrected-implementation table (Table 4) contradicts the "inherently stronger" narrative; Table 6 is misframed as a trade-off that isn't there; and the released repository is effectively empty — the source URL in the abstract contained no runnable material, forcing a full paper-only reimplementation with several critical documentation gaps (multi-bias SCM, lambda warm-up, kernel-bandwidth failure mode) surfacing along the way. The main results are plausible and the estimator itself checks out algebraically; the paper is not yet in a state where a reader can verify either the algebra or the empirical numbers.

Strengths

• Conceptual contribution. SAM provides a clean structural account of why conditional-independence-based fairness/robustness criteria work, giving sufficient pathway-blocking guarantees rather than the necessary-only versions in Veitch et al. (2021). Corollary 2.5 — folding mediator and backdoor biases into a single set $\mathbf{B}$ for the regularizer — is a genuinely useful practical simplification, modulo the caveat below about Selective Mediation.
• Estimator engineering. Turning conditional distance correlation from an $O(n^3)$-memory academic estimator into two backprop-compatible variants (subsampled DISCO$_m$ and single-shot sDISCO) is the kind of implementation-side contribution the field needs. The Diagonal Extraction Identity idea is elegant, and independent reimplementation confirms that sDISCO computes the intended $O(n^2)$ estimator (bit-for-bit against a reference naive implementation) once the missing algebra is filled in.
• Breadth of evaluation. The six-dataset testbed spans regression and classification, synthetic and real, and vision and NLP, which is broader than most recent bias-mitigation submissions. The pathway-counterfactual analysis on dSprites (Tables 3 and 6) complements the standard balanced-test-set evaluation and separates debiasing effectiveness from predictive retention.
• Confirmed causal-stability story. In an independent controlled experiment on dSprites, sDISCO$_{X,Y}$ nearly zeros counterfactual sensitivity on both spurious paths and attains the highest counterfactual accuracy, while sDISCO$_X$ (observing only one bias) collapses the observed path and leaves a clear residual on the unobserved one. This is exactly the theoretical prediction of Theorem 2.3 under partial observation, and it reproduces qualitatively even though absolute magnitudes differ.
• Honesty about limitations. Section 2.2 explicitly discusses positivity, unobserved confounding, and selective mediation; the FairFace ethical discussion is thoughtful; Appendix B openly reports that the previous sDISCO implementation was mathematically inconsistent and re-runs the affected experiments.

Weaknesses

• Central display equations are missing throughout. The definitions of ctf-stable, ctf-IE, ctf-SE (Section 2.1), the $V_{XY} = T_1 + T_2 - 2 T_3$ decomposition (Section 3.2 and Theorem G.1), the $D_3^{(i)}$ rewriting (Appendix G), and the counterfactual metrics $S_X$, $S_Y$, $\mathrm{Acc}_{\mathrm{ctf}}$, $R^2_{\mathrm{ctf}}$ (Sections 4.4 and H.4) are all announced by prose ending in a colon with no formula below. This is not a rendering nit: readers cannot check the TV-decomposition sign convention, cannot verify the sDISCO exactness claim, and cannot reproduce Tables 3 and 6.
• sDISCO grand-mean identity does not verify at index level. Writing $g^X = (W \circ (WA))\mathbf{1}$ gives entry $i$ equal to $\sum_j W_{ij} \sum_k W_{jk} A_{kj}$, which threads $W$ through a second row $j$, whereas the intended bilinear form $w^{(i)} A (w^{(i)})^\top$ uses only row $i$'s weights on both sides of $A$. Given Appendix B discloses that the earlier sDISCO version had exactly this kind of math–code mismatch, the exactness claim needs a line-by-line index-level verification (or an amended identity).
• Notation instability and role-reversal in Sections 4.4 / H.3 / H.4. $X$ is globally the model input but is reused for a specific dSprites bias axis; $Y$ is globally the target but is reused for the vertical-position bias; and H.3 uses Scale as target while H.4 uses $Y$-position as target — under the same overloaded letters and both cross-referenced as "the Section 4 setup". The two experiments are easy to conflate.
• "Inherently stronger" correction narrative is not supported by Table 4. A bug fix aligns the implementation with intent, it does not inherently improve numbers. Table 4 itself shows the older, mathematically-inconsistent implementation was better on Blob (0.788 vs. 0.751); and on MNLI GDRO is bolded as best (0.755) with the old sDISCO at 0.730. The "inherently yields stronger empirical performance" claim overreaches.
• Two Table 2 copies disagree with each other and with the prose. Table 2 (Sec 4.3) and the pre-correction Table 4 (App B) have overlapping sDISCO rows with different numbers, different bolding, and different dagger annotations. The main-text prose ("sDISCO ranks second", "sDISCO performs best on MNLI") is only consistent with one of them.
• Table 7 caption contradicts the data — and DISCO$_m$ OOMs before Naive DISCO. The caption says naive tensorized DISCO "hits a memory bottleneck early", but Table 7 shows Naive DISCO surviving batches 64–768 and OOMing only at 1024 — later than c-MMD, Fishr, IRM, and GDRO. More strangely, DISCO$_m$ — the memory-efficient subsampled variant — OOMs earlier than Naive DISCO, undermining the DISCO$_m$ efficiency framing.
• Table 6 framed as a trade-off that isn't there. sDISCO has both lower $S_X$ (0.058 vs 0.064) and much higher $R^2_{\mathrm{ctf}}$ than HSCIC/CIRCE. HSCIC and CIRCE are Pareto-dominated on both axes, not trading debiasing for accuracy. The correct reading actually strengthens the paper's case.
• Consistency proof shortcuts. Proposition 3.5 conflates the continuous mapping theorem with Slutsky's, never states the strict-positivity requirement on the population conditional distance variances (needed for the ratio map to be continuous), and applies the WLLN to the average of $m$ local estimators that share the same Nadaraya–Watson weights and are therefore strongly dependent.
• Corollary 2.5 vs. Selective Mediation. Corollary 2.5 argues "no need to distinguish between mediated and spurious bias effects", but Section 2.2 immediately reintroduces the distinction (via $\mathbf{W}_{\text{stable}}$ and $\mathbf{B}' = (\mathbf{W} \setminus \mathbf{W}_{\text{stable}}) \cup \mathbf{Z}$). The corollary's prose is stronger than what has been shown.
• Empirical rigor. No paired-seed significance tests, no per-seed rankings, and no bootstrap CIs are reported despite several headline advantages sitting inside one standard deviation (e.g. FairFace: DISCO$_m$ 0.860 ± 0.006 vs. GDRO 0.840 ± 0.012). CIRCE's Waterbirds win is quietly explained away by a larger search budget, but DISCO's performance under a matched budget is not reported. There is also no sensitivity ablation for $\sigma_Y$ or batch size, both of which control this per-batch estimator's behaviour.
• Positivity is stated but never checked on the actual splits. The theory hinges on $P(B \mid Y) > 0$, but the biased splits are engineered with strong shortcuts (Waterbirds ≈95% background–bird alignment, YaleB quantile-based selection) and the empirical counter-example rate is never reported.
• Scope and positioning. "Fewer hyperparameters" is really only true against HSCIC and CIRCE; c-MMD and Adversarial match DISCO's count. The paper also does not situate itself against the unsupervised-bias family (LfF, JTT, DFR, Correct-N-Contrast), which is the natural contrast for anyone who cannot label $\mathbf{B}$.
• Load-bearing proof steps are asserted, not walked through. The d-separation $\hat{Y}_{y', \mathbf{w}'} \perp \mathbf{W}_{y''}, \mathbf{W}, Y \mid \mathbf{Z}$ is read off Fig. E.2 without explicitly enumerating the blocked paths in the parallel-worlds graph; Prop E.1's converse-failure claim is delegated to a cited work; Theorem F.7's proof phrases barycenter injectivity as if it "automatically follows" from strong negative type rather than being equivalent to it (Lyons Lem. 3.9).
• Minor prose defects. The Unobserved Confounding paragraph contains a scrambled sentence ("forremains", "ofthese") that garbles the exact boundary condition. The introduction abbreviates "spurious indirect effects (IE)" while Section 2 keeps IE and SE strictly separate. Table 5 contains unlabelled rows with different $\lambda$ grids.

Reproducibility & code

• Released repository shipped no runnable material. The abstract advertises a source-code URL and Appendix B references the corrected sDISCO implementation as the source of Table 2, but the repository we could inspect contained a stub README and nothing else — no training scripts, no config files, no benchmark harness, no baseline reimplementations, no sDISCO source. 100% of the executed replication code is a reimplementation from the paper's equations, not the authors' own. This is the largest single reproducibility gap.
• Headline aggregates only partially replicate — mostly due to coverage. Of Table 2's six datasets, three (dSprites $R^2$, Blob $R^2$, a reduced FairFace) were reproduced under a paper-only reimplementation; YaleB and Waterbirds require manual non-HuggingFace downloads that could not be obtained under the time budget, and the MNLI TinyBERT backbone ships only a .bin checkpoint incompatible with a stable torch+CUDA stack. Where numbers did land, the synthetic-$R^2$ orderings broadly hold (sDISCO 0.751 / DISCO$_m$ 0.734 on dSprites vs paper 0.687 / 0.688; sDISCO 0.818 on Blob vs paper 0.751); FairFace, run at reduced scale to fit shared-cluster memory, fell to 0.64 for DISCO — just below the replicated biased backbone (0.653) — a scale artifact rather than a contradiction, but it means the FairFace headline cannot be considered independently verified.
• Pathway-counterfactual result reproduces qualitatively but not quantitatively. The Table 3 pattern reproduces cleanly (sDISCO$_{X,Y}$ nearly zeros both $S_X$ and $S_Y$ and attains the highest $\mathrm{Acc}_{\mathrm{ctf}}$; sDISCO$_X$ zeros the observed path and leaves residual on the unobserved one; ResNet is worst on all three). The sDISCO$_{X,Y}$ row lands inside the paper's 0.03–0.05 band ($S_X$ 0.025 vs 0.066, $S_Y$ 0.028 vs 0.067, $\mathrm{Acc}_{\mathrm{ctf}}$ 0.961 vs 0.98), but ResNet magnitudes (0.192 / 0.189 / 0.739) are substantially lower than the paper's (0.302 / 0.275 / 0.65) and sDISCO$_X$ residuals also diverge (0.021 / 0.103 vs 0.117 / 0.154), consistent with the missing $S_X$ / $S_Y$ formulas admitting several natural interpretations.
• Table 6 story reproduces stronger than the paper reports. Under reimplementation sDISCO again dominates ($S_X$ 0.063, $R^2_{\mathrm{ctf}}$ 0.852) with HSCIC (0.099 / 0.789), CIRCE (0.097 / 0.795) and Adversarial (0.270 / −0.284) all worse on both axes — the correct Pareto-dominance reading rather than the trade-off framing in the manuscript.
• Runtime table's scaling narrative reproduces in structure, not in per-cell OOM boundaries. Under a reimplementation harness sDISCO scales to batch 1024 while both DISCO$_m$ and Naive DISCO OOM at large batches and per-step time is on par with the fastest baselines — the qualitative $O(n^2)$-vs-$O(n^3)$ story. But the exact OOM boundaries do not match: DISCO$_m$ OOMs at 1024, Naive DISCO at 768 (paper has these reversed), and the differential only survives under a stripped-pipeline harness; a full-pipeline run OOMs every method at 768.
• Paper under-specification surfaced during replication. The multi-bias dSprites SCM in Appendix H.1 produced inverted results on a literal reading (naive ResNet was already causally stable) and had to be reconstructed so that the positional shortcut perfectly predicts the noisy label before the paper's shortcut-learning ordering appeared; sDISCO collapsed to a constant output at full penalty strength and required a linear $\lambda$ warm-up not described in the paper; and a too-large kernel bandwidth silently turns the conditional penalty into an unconditional one, again causing collapse with no diagnostic. These are documentation gaps, not method defects, but they are the difference between a working reimplementation and one that quietly fails.
• Selected hyperparameters and training regime are not disclosed. Table 5 lists the grids searched but not the winning $(\lambda, \mathrm{bw})$ per (method, dataset). Combined with the missing optimizer / LR schedule / epoch count / seed count / batch-size specification, hitting Table 2 values within tolerance from the paper alone is infeasible.
• Custom small ResNet, label noise, FairFace binarization, Fishr/IRM group construction, and Table 5 unlabelled rows are all under-specified — each in ways that meaningfully affect the reported numbers, as detailed in the anchored comments.

Recommended Changes

Essential

• Release runnable code and configs. Populate the repository referenced in the abstract with (i) the sDISCO / DISCO$_m$ implementation used for Table 2, (ii) the seven baseline reimplementations (especially the group-based IRM/Fishr and the adversarial three-step schedule), (iii) per-(method, dataset) config files with the selected hyperparameters that produced the reported means, and (iv) a benchmark harness. This addresses the "released repository is empty" weakness and is the single most consequential change: without it, none of Tables 2, 3, 6, 7 can be reproduced.
• Restore the missing display equations. Reinsert the formulas for ctf-stable, ctf-IE, ctf-SE (Section 2.1), the $V_{XY} = T_1 + T_2 - 2 T_3$ decomposition with explicit $T_1, T_2, T_3$ (Section 3.2 and Theorem G.1), the $D_3^{(i)}$ rewriting step (Appendix G), and the counterfactual metrics $S_X$, $S_Y$, $\mathrm{Acc}_{\mathrm{ctf}}$, and $R^2_{\mathrm{ctf}}$ (Sections 4.4 and H.4). All are load-bearing for the corresponding weaknesses.
• Verify the sDISCO grand-mean identity at the index level. Show a two-line index derivation that $[g^X]_i = w^{(i)} A (w^{(i)})^\top$, either confirming $(W \circ (WA))\mathbf{1}$ is that quantity (with the appropriate transpose / symmetrization) or supplying the corrected identity. This directly addresses the largest technical concern flagged in Weaknesses.
• Report the training regime, selected hyperparameters, label-noise scale, and lambda warm-up. For each (method, dataset), state: optimizer, LR + schedule, epochs, batch size, number of seeds, and the winning $(\lambda, \mathrm{bw}, \dots)$. Also state the label-noise distribution and scale per dataset and describe the $\lambda$ warm-up schedule needed to keep sDISCO from collapsing.
• Specify the multi-bias dSprites SCM explicitly. State how the positional shortcut is coupled to the noisy target so that the shortcut-learning ordering in Section 4.4 reproduces on a literal reading of Appendix H.1.
• Reconcile the two copies of Table 2 and remove the "inherently yields stronger performance" claim. Keep only the corrected numbers as Table 2, keep the old values as Table 4 for transparency, and soften Appendix B to "our corrected implementation yielded stronger empirical performance in our experiments", given that Table 4 shows the old numbers dominate on at least Blob.
• Fix the notation collision and role-reversal between Sections 4.4 / H.3 / H.4. Rename the dSprites bias axes to something like $B_{\mathrm{x\text{-}pos}}$ / $B_{\mathrm{y\text{-}pos}}$, and state explicitly at the top of each of Section 4.4, H.3, and H.4 what the target and biases are for that experiment.
• Rewrite the garbled Unobserved Confounding sentence. The current sentence containing "forremains" and "ofthese" is scrambled and is the operational boundary condition of the guarantee — restore it as a clean statement.

Suggested

• Repair the DISCO$_m$ consistency proof. Replace "Slutsky's theorem" with the continuous mapping theorem, state the strict-positivity condition on the population conditional distance variances, and either give a proper double-limit ($n \to \infty$ then $m \to \infty$) or add a weak-dependence argument for the average of shared-weight local estimators.
• Reconcile Table 7 with its caption and explain the DISCO$_m$ vs Naive DISCO inversion. State the value of $m$ used per batch size, either qualify the "naive DISCO hits a memory bottleneck early" claim to match the data or explain the discrepancy, and clarify why the subsampled DISCO$_m$ OOMs earlier than the naive tensorized version.
• Rewrite the Table 6 discussion as Pareto dominance. Frame HSCIC/CIRCE as dominated by sDISCO on that experiment rather than as trading debiasing for accuracy — the corrected reading actually strengthens the paper's case.
• Soften Corollary 2.5 or reconcile with Selective Mediation. Replace "no need to distinguish" with "once the practitioner has separated stable from unstable mediators, the remaining $\mathbf{W} \cup \mathbf{Z}$ can be treated uniformly".
• Match hyperparameter budgets or report a sensitivity. For Waterbirds specifically, either equalize CIRCE's search budget with DISCO's or show DISCO's performance under CIRCE's larger grid.
• Add paired-seed significance tests or bootstrap CIs, especially where headline advantages fall inside one standard deviation (FairFace, YaleB), and report per-seed rankings to substantiate "consistently outperform".
• Add a bandwidth × batch-size ablation on at least one dataset, since sDISCO is a per-batch statistic.
• Verify positivity on the biased splits. Report the empirical $P(B \mid Y)$ tail per dataset and a small ablation of DISCO performance as overlap shrinks.
• Report FairFace binarization counts and, if possible, a sensitivity check under a finer skin-tone coding, so the "dark = Black, light = others" collapse is auditable.
• Specify Fishr/IRM group construction per dataset and explain the "–" entries on regression tasks; specify the small ResNet used on Blob to a level that enables reconstruction.
• Attribute each row of Table 5 to a specific method (and dataset if applicable), so the extra unlabelled $\lambda$ grids are unambiguous.
• Rephrase "fewer hyperparameters" as "fewer than HSCIC/CIRCE" rather than a blanket claim, since c-MMD and Adversarial match DISCO's count.
• Discuss unsupervised-bias baselines. Even a paragraph situating DISCO against LfF / JTT / DFR / Correct-N-Contrast would clarify the scope of the informed-bias contribution.
• Rescue MNLI's selection protocol with an independent WGA validation slice, or explicitly justify the selection-on-target-metric approach given the rest of the paper uses unbiased test sets.
• Provide an explicit two-line SCM counterexample for Prop E.1's converse-failure claim rather than delegating to Plecko & Bareinboim.
• Rewrite the load-bearing d-separation step in the twin graph by explicitly enumerating the paths in Fig. E.2 blocked by conditioning on $\mathbf{Z}$, and rephrase Theorem F.7's proof so that barycenter injectivity is derived from strong negative type (Lyons Lem. 3.9) rather than said to "automatically follow" from the chosen embedding.
• Sketch why the constrained MLE in Theorem 2.4 still maximizes ctf-stable, since the unconstrained MLE that "implicitly maximizes TV" need not coincide with the constrained one.