A Random Matrix Perspective on the Consistency of Diffusion Models

Summary

The paper studies the observation that diffusion models trained on non-overlapping data splits map the same noise seed to nearly-identical images, and argues that this consistency is largely a linear phenomenon inherited from the shared Gaussian statistics of the data. The conceptual move is to analyse the linear Wiener denoiser under finite-sample randomness of the empirical covariance $\hat{\boldsymbol\Sigma}$, using deterministic-equivalence (DE) tools from random matrix theory. The main analytical results are: (i) in expectation, finite data act by renormalizing the noise level, $\sigma^2 \mapsto \kappa(\sigma^2)$, where $\kappa$ solves the Silverstein self-consistent equation, and this explains overshrinkage of low-variance modes; (ii) across dataset realizations, the denoiser variance factorizes into an anisotropy term in the probe direction $\mathbf{v}$, an inhomogeneity term in the noised input $\mathbf{x}$, and a global $n$-scaling; and (iii) an extension of DE to fractional matrix powers (via a Balakrishnan integral representation) makes the same analysis go through for the full Wiener sampling map, not just the single-step denoiser. Deep-net validation on UNet and DiT across FFHQ, CIFAR, AFHQ and LSUN identifies a memorization$\to$renormalization transition, and shows that RMT-predicted per-seed rank-orderings correlate with cross-split MSE (Spearman $\approx 0.33$). The framework is technically substantive: extending DE to fractional powers is a genuine technical contribution, and the anisotropy/inhomogeneity decomposition provides a principled spatial and spectral map of where diffusion generations disagree. The main conceptual limitation is that the linear surrogate is qualitative for deep networks — the predicted rank correlation is modest and absolute magnitudes are architecture-dependent — and several Gaussian-data assumptions used inside the derivations are not clearly delineated. The manuscript would benefit from tighter statistical framing of the deep-net validation and from repairing scrambled statements around the central sampling-map propositions.

Strengths

• Conceptual contribution. Isolating the finite-sample noise renormalization $\sigma^2 \mapsto \kappa(\sigma^2)$ as the source of overshrinkage, and using it to explain why low-variance eigenmodes require the largest $n$ to become consistent, gives a mechanistic story with clear falsifiable content.
• Technical contribution. Extending deterministic equivalence to fractional matrix powers (Prop. C.6-C.8) via a Balakrishnan integral representation is a novel and useful tool that lets the Wiener sampling map be handled with the same DE machinery as the single-step denoiser.
• Variance decomposition. The factorization into anisotropy $\chi(\lambda_k, \kappa) = \lambda_k/(\lambda_k+\kappa)^2$, inhomogeneity in $\bar{\mathbf{x}}$, and a $1/n$ global scaling is interpretable and generates the sharp predictions that low-variance modes need more data to align, and that top-eigenspace-aligned initial noise induces more cross-split disagreement.
• Empirical breadth. Validation is not one-dataset — UNet/DiT sweeps on FFHQ32/64, CIFAR10/100, AFHQ32, and LSUN church/bedroom at 32 and 64 pixels give the qualitative claims broader support than a typical theory paper provides.
• Counterfactual control. The PC2-stratified split experiment (Figs. 13-14) is a clean falsification test: intentionally mismatched moments produce visibly less consistent generations, tightening the causal reading of the Gaussian-statistics story.
• Theory-side reproducibility scaffolding. The released rmt_core module (Silverstein $\kappa$ solver with Newton continuation, and the tangent-mapped Gauss–Legendre quadrature for the DE integrals) is a genuinely usable numerical backbone for the analytical curves in Figs. 2, 4 and 16.

Weaknesses

• Modest deep-net effect size vs. strong framing. The paper's headline linear-theory-vs-deep-net bridge is a Spearman correlation of $\approx 0.33$ on FFHQ64 UNet ($n=30{,}000$), yet the surrounding prose reads as if architecture were irrelevant to consistency. Fig. 5C, however, shows DiT is systematically more consistent than UNet at every $n$, so the linear theory captures a rank-ordering but not the magnitudes.
• Gaussian-data assumptions are baked into 'sharp' predictions. The ellipsoidal-shell argument that produces the inhomogeneity scaling $\sqrt{\sigma^2 + \lambda_k}$ presumes $\mathbf{x} - \boldsymbol\mu$ is Gaussian; images are not. The paper does not delineate which of the fluctuation predictions inherit this assumption strictly and which survive up to fourth-order corrections.
• Silent switch between population and empirical shells. In the fluctuation derivation, one line says the noised sample is $\mathcal{N}(\boldsymbol\mu, \hat{\boldsymbol\Sigma}+\sigma^2 I)$ and the next places the probe on the shell $\mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma + \sigma^2 I)$. Since the whole point of the RMT analysis is that $\hat{\boldsymbol\Sigma}\neq\boldsymbol\Sigma$ at finite $n$, this substitution deserves a formal justification.
• Mean-fluctuation term is assumed away. The $\hat{\boldsymbol\mu} = \boldsymbol\mu$ assumption isolates the covariance effect but the neglected term has variance of order $\boldsymbol\Sigma/n$, i.e. the same order as the fluctuation the paper analyses; a scaling argument or numerical control is missing.
• Scrambled key statements. The Prop. 5.1 preamble that defines $\bar{\mathbf{x}}$ and the $\sigma_T\to\infty$ limit is grammatically corrupted, and the Fig. 4 B/C caption mixes 'input location' with 'probe direction $\bar{\mathbf{x}}$' plus a stray '$\mathbf{v}$, with larger deviation on top eigenspaces' fragment. These are close to the paper's central sampling-map claims and currently unreadable.
• Notation collision in Prop. 5.2. The proposition uses $\mathbf{v}$ as both the probe direction (in $\mathbf{v}^\top S_D(\mathbf{x})\mathbf{v}$) and as a scalar noise-scale integration variable via $\kappa'\!:=\!\kappa(v^2)$; this needs disambiguation.
• 'Vanishing' at $n\sim 30{,}000$ is stronger than the theory predicts. For FFHQ64, $d=12{,}288$, so $\gamma\approx 0.4$ at $n=30{,}000$; the Silverstein equation still gives non-negligible $\kappa(\sigma^2)>\sigma^2$. The wording 'vanishes when learned and population spectra coincide' overreads the theory.
• $\sigma_0=0.002$ numerical floor confounds the low-eigenmode overshrinkage evidence. Fig. 4A / 16A is the primary numerical support for Prop. 5.1, and the appendix acknowledges the low-eigenmode region is precisely where a $\sigma_0$-floor artefact appears; the artefact's sign happens to protect the qualitative conclusion, but its magnitude is not quantified.
• Weakly-quantified motivating claim. 'Samples closer to the Gaussian solution are also more consistent across splits (Pearson $r=0.244$)' explains $\approx 6\%$ of variance; the phrase 'underlies consistency' is a strong reading of a small effect.
• Only two independent splits per dataset size. The reported 'mean $\pm$ std' in Fig. 5C summarises seeds inside a single split pair, not variance across independent split realizations, so it does not directly bracket the sample-covariance-fluctuation contribution.
• Two-phase boundaries. Memorization is defined as $n\le 1000$ and renormalization as $n\ge 3000$; the $1000 < n < 3000$ interval is not classified even though the phase story frames the whole Sec. 6 narrative.
• Local proof gaps. The claim that $f(\lambda_k)$ peaks at $\lambda_k = \sqrt{\sigma^2\kappa}$ with value $(\sqrt{\kappa}-\sigma)/(\sqrt{\kappa}+\sigma)$ is stated without the explicit closed form of $f$, and the summary paraphrases this as 'around $\sigma^2$ and $\kappa$', which is not equivalent to the geometric-mean location; the ' $\kappa > \sigma^2$' transitions from the derived $\ge$ to strict $>$ without noting the operating conditions.

Reproducibility & code

• Theory-side pipeline works. rmt_core (the Silverstein $\kappa$-solver solve_kappa, SpectrumKappa_np/_mpmath, and the 2D Gauss–Legendre integrator on the tangent-mapped domain) implements all deterministic-equivalence integrals in the paper (Eqs. 7, 9, 10, 25, 27, 31). Given any spectrum, Fig. 2B and the analytical curves in Figs. 4/16 are recoverable. docs/data/spectrum.json contains a synthetic power-law spectrum sufficient for the qualitative $\kappa(\sigma^2)$ story, so this piece of the paper is 'supported' at the analytical level.
• DNN-side pipeline is missing. There is no training script for UNet-CNN or DiT, no split-construction code, no trained checkpoints, no sampler (the described 35-step Heun sampler is not implemented in the repo), and no image datasets or preprocessing pipeline for FFHQ, AFHQ, CIFAR or LSUN. All quantitative DNN figures — Fig. 1B (block MSE heatmap over 512 seeds), Fig. 5B/C/E/F (memorization threshold, cross-split MSE curves, per-eigenmode decomposition, Spearman $r=0.33$), Figs. 22-32 (multi-dataset spectral decompositions), Figs. 33-35 (per-seed inhomogeneity), and Figs. 36-38 (noise-space perturbations) — depend on this missing pipeline.
• Linear-denoiser validation figures require missing preprocessing. Fig. 15 (denoiser Pearson $r$ across noise scales) and Fig. 16 (sampling-map scaling and $1/n$ fit) are within reach of rmt_core primitives once the FFHQ32 image tensor and the specific $n=1000$ subset are supplied, but neither the preprocessing pipeline nor the subset seed is documented.
• Counterfactual control lacks scripts. The PC2-stratified partitioning (Figs. 13-14) — an experiment that carries substantial rhetorical weight — has no partitioning code, no training driver, and no numeric tables; only precomputed thumbnails appear in docs/assets/counterfactual/.
• Cross-split MSE table for Fig. 1B not shipped. The heatmap is the paper's motivating quantitative evidence and no aggregation script exists; the underlying 512-seed MSE tensor is not distributed.
• No hint of the FFHQ64 Spearman-correlation artefacts. The theory-side of Fig. 5F is implementable, but the per-seed empirical MSE vector needed to compute the correlation is not available in the repo.

Recommended Changes

Essential

• Repair the corrupted Prop. 5.1 preamble. Re-typeset the sentence around 'we denote the shift and normalized noise $\bar{\mathbf{x}}$' so that the definition and the $\sigma_T\to\infty$ limit statement are separated and readable; this is the direct set-up for the paper's main sampling-map proposition.
• Fix the Fig. 4 B/C caption. Panel B is anisotropy in $\mathbf{v}$ and panel C is inhomogeneity in $\bar{\mathbf{x}}$; the current caption interleaves both variables and leaves a stray fragment ('v, with larger deviation on top eigenspaces') that appears to belong elsewhere.
• Rename the scalar variable in Prop. 5.2 so it no longer collides with the probe-direction $\mathbf{v}$; e.g. use $\kappa'=\kappa(w^2)$ with $u,w$ as the two integration variables.
• Qualify 'no knowledge of network architecture'. The narrow claim (RMT predicts rank-ordering of per-seed deviations) is correct, but Fig. 5C shows DiT is uniformly more consistent than UNet, so add a clause acknowledging that architecture controls the magnitude of cross-split MSE even when it does not control the spatial pattern.
• Replace 'vanishes at $n\sim 30{,}000$' with 'empirically indistinguishable at $n\sim 30{,}000$' and, if space permits, report the theory-predicted residual overshrinkage at $\gamma\approx 0.4$ so readers can see how much of the small remaining deviation is expected.
• Justify the population-vs-empirical shell switch in the inhomogeneity derivation, either as a leading-order approximation in $\gamma$ or by redoing the ellipsoidal-shell argument with $\hat{\boldsymbol\Sigma}$ throughout.
• Justify $\hat{\boldsymbol\mu}=\boldsymbol\mu$. Add a short lemma or a numerical control showing the neglected mean-fluctuation term is subleading in the operating $\gamma$, since it is otherwise of the same order as the covariance-only fluctuation the paper analyses.
• Release the training/sampling pipeline and split hashes used for the UNet/DiT experiments (Section 6 and Appendix D.3), so that Fig. 1B, Fig. 5, and the multi-dataset appendix figures become independently reproducible; also ship the aggregation scripts for Fig. 1B and the per-seed MSE table for Fig. 5F.
• Document dataset preprocessing. Add a data/ README covering FFHQ32/64, AFHQ32, CIFAR10/100 and LSUN church/bedroom at 32 and 64 pixels (source, crop, resize, normalization, subset seed) so that the fitted scaling exponents (e.g. $\alpha\approx-0.84$ in Fig. 16B) can be independently reproduced.

Suggested

• Quantify the $\sigma_0=0.002$ artefact in Fig. 4A / 16A, or overlay the $\sigma_0=0$ theory curve floored at $\sigma_0$, so the low-eigenmode overshrinkage evidence is not entangled with the numerical wall.
• Soften 'underlies consistency'. Report the $R^2$ associated with $r=0.244$ or reframe the sentence as a rank-order association rather than a causal statement.
• Report an intermediate dataset size (e.g. $n=2000$) to close the gap in the two-phase description between memorization ($n\le 1000$) and renormalization ($n\ge 3000$).
• State the $L^2$-optimality condition for the moment-matched Gaussian score, or downgrade 'usually the best linear approximation' to 'the moment-matched linear approximation'.
• Delineate Gaussian-data scope. Add a sentence in Sec. 4.3 or Sec. 6 identifying which fluctuation predictions rely strictly on Gaussianity of the noised data and which are expected to be robust to fourth-order corrections.
• Add per-split-realization variance. At one representative $n$, train at least a second independent pair of non-overlapping splits and report the additional variability so that the 'mean $\pm$ std' in Fig. 5C brackets across-split variation, not only within-split seed variation.
• Show the closed form of $f(\lambda_k)$ and the elementary maximization step underlying the $\sqrt{\sigma^2\kappa}$ peak location and the $(\sqrt{\kappa}-\sigma)/(\sqrt{\kappa}+\sigma)$ peak value; then reconcile the summary paraphrase ('around $\sigma^2$ and $\kappa$') with the sharper geometric-mean characterisation.
• Distribute the PC2-stratification and counterfactual training scripts so that the moment-manipulation experiment (Figs. 13-14) can be reproduced, and release the perturbation script and (ideally) one FFHQ64 checkpoint for Figs. 36-38.
• Clarify the Kamb & Ganguli attribution in Appendix A with a one-clause distinction between local coherence (mosaic) and global layout (attention), to resolve the apparent internal tension.