SVRG and Beyond via Posterior Correction

Summary

The paper draws a conceptual bridge between SVRG-style variance reduction and Posterior Correction (PoCo), a recent Bayesian mechanism for knowledge adaptation. The central move is that the SVRG inner-loop correction $\nabla \ell_i(\theta_{\text{in}}) - \nabla \ell_i(\theta_{\text{out}}) + \mathbf{g}_{\text{out}} / N$ can be derived as the deterministic ($\varepsilon \to 0$) limit of a natural-gradient PoCo update over an isotropic-Gaussian posterior; the induced VSGD-PoCo update matches Alg. 1 exactly under the delta approximation (Thm. 2). Building on this identification, the authors substitute richer exponential-family posteriors to obtain algorithms that would not fall out of the classical SVRG derivation: VON-PoCo, a full-Gaussian Newton-like variant that adds a Hessian correction ("SVRH"), and IVON-PoCo / IVON-PoCoMo, a diagonal, Adam-like scaling that inherits the outer/inner structure but tolerates mega-batches. Empirically, IVON-PoCo cleanly matches an L-BFGS reference on convex logistic regression where IVON alone stalls, with characteristic accuracy jumps at every mega-batch refresh — a compelling proof of concept. At deep-learning scale the picture is honestly reported as mixed: IVON-PoCoMo improves final GPT-2 validation PPL vs. IVON and AdamW when counted in optimizer steps, but under matched gradient computations or wall-clock is indistinguishable from IVON with a $3\times$ larger batch; ResNet-50 / ImageNet shows the same pattern. The reframing is genuinely useful — it links two literatures and yields a principled recipe for Hessian correction — but three limitations weigh against the strongest version of the contribution: the Newton-like VON-PoCo, which is the clearest "beyond SVRG" claim, is never experimentally evaluated; no convergence theory is offered for any of the new algorithms; and under the mega-batch regime actually used, the isotropic-Gaussian diagonal case reduces to $\alpha$-SVRG, so the practical empirical delta from prior work rests only on the Hessian correction, whose deep-learning benefit is not yet established. Independent execution of the release reproduces the qualitative IVON-PoCoMo $>$ IVON ordering on a small Shakespeare GPT-2 run and the IVON baseline on CIFAR-10 ResNet-20, but the abstract's marquee results — logistic regression and GPT-2 pretraining — have no counterpart in the shipped code and could not be re-verified.

Strengths

• Conceptual contribution. The identification of SVRG (in the delta limit) with an isotropic-Gaussian PoCo update is elegant, previously unnoticed, and connects two mostly disjoint literatures (variance reduction and Bayesian knowledge adaptation). The reinterpretation of gradient corrections as knowledge transfer between old and new gradients is genuinely new.
• Principled recipe for extensions. Once the equivalence is set up, the Newton-like SVRH pre-conditioner and the diagonal IVON-PoCo fall out mechanically from choices of $q$. This is the kind of unification-then-derivation that Bayesian-learning-rule-style work is at its best doing.
• Honest deep-learning results. The paper is admirably direct about the negative wall-clock result on GPT-2 (matching Defazio & Bottou 2019), does not oversell ImageNet, and provides ablations (starting time of correction, mega-batch factor, $\alpha$ schedule) that reveal rather than hide sensitivity.
• Convex demonstration. The logistic-regression figures show the mechanism working exactly as advertised — big performance jumps at each mega-batch refresh, convergence to the L-BFGS reference — which is the clearest evidence that the framework is not merely notational.
• Practical algorithm. IVON-PoCoMo integrates the corrections into a scalable IVON-style Adam-like optimizer with mega-batching, momentum, weight decay, and $\alpha$-downweighting — enough engineering care that the method is actually runnable at LLM scale.

Weaknesses

• Newton variant is theoretical only. VON-PoCo (Alg. 5) is the mechanism the paper cites as clearly "beyond SVRG," yet it is never used in any experiment — every reported result uses the diagonal IVON-PoCo. The Newton-with-Hessian-correction story therefore has no empirical support even on the convex problems where it would be tractable to demonstrate.
• No convergence theory. Every cited SVRG variant (SAG, SAGA, SARAH, SPIDER, $\alpha$-SVRG, …) comes with rate guarantees. This paper offers none for VSGD-PoCo, VON-PoCo, or IVON-PoCo, and does not discuss whether Thm. 2's equivalence transfers Johnson–Zhang guarantees or how the delta approximation, mega-batching, and Hessian correction affect them.
• Isotropic case reduces to $\alpha$-SVRG in practice. The paper acknowledges that the mega-batched, isotropic-Gaussian, $\alpha$-downweighted variant reduces to $\alpha$-SVRG (Yin et al., 2025). Since every deep-learning experiment uses mega-batches, the delta from $\alpha$-SVRG is essentially just the Hessian correction — a much narrower empirical contribution than the abstract's framing suggests.
• Hessian-correction novelty is overstated. The paper claims most Newton-SVRG variants "do not do any Hessian correction at all," but the next paragraph acknowledges that Gower et al. (2018, SVRG2) and Chayti et al. (2024) already use terms of the form $\mathbf{H}_{\text{out}\setminus i}(\mathbf{m}_{\text{in}} - \mathbf{m}_{\text{out}})$. The correct claim is that SVRH is structurally different (expected outer Hessian averaged over $q_{\text{out}}$), not that Hessian corrections are novel.
• "Special case" framing for Thm. 2. SVRG is recovered only in the $\varepsilon \to 0$ (delta) limit; at any positive noise VSGD-PoCo is a different algorithm — and Fig. 3 suggests the added sampling gives no benefit over SVRG in the isotropic case. The abstract's "SVRG can be recovered as a special case" understates the approximation.
• No matched-compute deep-learning speed-up, yet framed as "acceleration." The abstract and Sec. 4 use verbs like "accelerates," "boosts," and "scales," but Fig. 4b shows IVON-PoCoMo does not beat IVON at $3\times$ batch size under matched compute, and ImageNet gains disappear when counted in data examples seen. The framing should be recalibrated to match this honest empirical picture.
• No seed variance for headline claims. The 17.4 / 18.0 / 18.4 GPT-2 PPL numbers, the ImageNet accuracy curves, and the Table 1 finetuning gains are all single-run. Gaps of 0.5–1 pp / 0.6 PPL at these scales are within seed noise; no CI, repeats, or significance tests are reported.
• Table 1 gains are within noise. Improvements of 0.0–0.9 pp across ViT/Qwen/Llama benchmarks, with $\sim 2\times$ the runtime and no seed averaging, do not establish that IVON-PoCo is better than IVON for finetuning.
• Ablation scope is narrow. The $\alpha$-value, $\alpha$-schedule, and inner-loop-size ablations all use a 33M wikitext103 model — not the 125M GPT-2 or ImageNet models where the recommendations are then implicitly applied. The paper's advocacy of an increasing $\alpha$ schedule (against $\alpha$-SVRG's decreasing one) rests on this small-scale evidence only.
• New hyperparameters have no sensitivity study. IVON-PoCoMo's outer-momentum coefficients $\rho_1$ and $\rho_2$ take three different pairs across the three experiments that use them, with no ablation showing how sensitive final results are to the choice.
• Algorithm 4 has notation drift. Line 1 initializes h but defines $\sigma_{\text{in}}$ in terms of h_in; $\rho_1$ and $\rho_2$ are called a singular "outer loop learning rate"; the text says to down-weight "the extra terms in line 17 of Alg. 4" but line 17 is end for. The renderings of lines 7–10 also mix multiple assignments into a single dense expression that is hard to parse.
• Narrow convex baseline suite. The only convex benchmark is logistic regression on MNIST/Covertype/CIFAR-10 flattened features — a very small slice of the ridge / SVM / matrix-completion problems on which classical SVRG variants are typically evaluated, and the Newton variant (where the Hessian correction should matter most) is not evaluated on any of them.

Reproducibility & code

Execution and inspection of the shipped release cover only two of the paper's experiment stacks — CIFAR-10 ResNet-20 image classification and GPT-2 (Shakespeare + OpenWebText) pretraining. Where code exists it is internally consistent with the paper's algorithms and, when executed, matches the paper's qualitative story on the small-scale runs the environment permitted; the shipped defaults, however, don't reproduce the paper's numbers without edits, and four of the eight main empirical claims have no corresponding code at all.

• Headline logistic-regression results (Figs. 2 & 3) are un-runnable. No SGD/SVRG/VSGD/VSGD-PoCo/IVON/IVON-PoCo logistic-regression script, no MNIST/Covertype loader configured for logistic regression, and no L-BFGS reference ships. This is the cleanest empirical demonstration in the paper and cannot currently be reproduced from the release.
• CIFAR-10 ResNet-20 partially reproduces but defaults don't match the paper. With edits to bump the shell-script default from 20 → 100 epochs and reduce dataloader workers, the IVON baseline reproduced cleanly (90.48% vs. paper's 90.46%). The IVON-PoCoMo leg was progressing normally but ran ~3× slower per epoch and was killed before completion, so the decisive 91.21% > 90.46% ordering (Fig. 6) could not be confirmed end-to-end in this environment; the shipped script also uses $\text{corralpha}=0.4$, $\rho_1=\rho_2=1.0$, $\text{ess}=5 \times 10^4$, whereas App. F.1 uses $\alpha=0.1$, $\rho_1=\rho_2=0.3$, $\text{ess}=5\times 10^5$.
• Shakespeare GPT-2 smoke test reproduces the qualitative claim. Running the shipped config/shakespeare_ivon.py and config/shakespeare_ivon_pocomo.py unchanged produces best validation losses of 1.4840 (IVON) vs. 1.4624 (IVON-PoCoMo), within $\sim 0.01$ of the repo's reference 1.4733 / 1.4577. This is not one of the paper's main claims but is the only end-to-end positive evidence that the shipped IVON-PoCoMo code implements the intended mechanism.
• GPT-2 OpenWebText run was infeasible in this environment; baselines are also missing. The three 100k-iter multi-GPU multi-day runs plus a ~54 GB tokenized download could not be executed here, so the 17.4 / 18.0 / 18.4 PPL headline numbers were never re-verified. Even if compute were available, the shipped repo has no AdamW baseline config (for the 18.4 figure) and no "IVON with $3\times$ batch size" config (for Fig. 4b's compute-matched panel).
• ImageNet ResNet-50 pipeline is absent. No ImageNet dataloader is registered, no 8-GPU DDP launcher, no SGD/AdamW/IVON/IVON-PoCo config for ImageNet; Fig. 5 cannot be reconstructed from the release.
• $\alpha$-SVRG baseline is not implemented. Yin et al. (2025) is the closest existing method to IVON-PoCo and the reference baseline in Figs. 6, 7, 8b, and 9; the release ships no $\alpha$-SVRG optimizer, so none of those comparison curves are reruntable.
• Finetuning pipelines (Table 1) are absent. No ViT-B/32, Qwen2.5, or Llama-3.1 LoRA finetuning code; no Cars/DTD/GTSRB/RESISC45/XSUM/GSM8k loaders or evaluation harnesses (ROUGE, GSM8k accuracy).
• Wikitext103 ablations and continual pretraining are absent. No 33M GPT-2 config (n_layer=4, n_head=4, n_embd=512), no wikitext103 loader, no wall-clock-budget sweep, and no FineWeb-edu continual-pretraining config; App. E.2 / E.4 experiments cannot be reproduced.

Recommended Changes

Essential

• Add experiments for VON-PoCo (Alg. 5). Run VON-PoCo on the same convex logistic-regression benchmarks used in Sec. 4.1 and compare to gradient-only Newton-SVRG variants; this is needed to substantiate the "Newton-like extension with Hessian corrections" claim flagged in Newton variant is theoretical only.
• Report seed variance for headline results. Provide at least two seeds (or a paired data-order re-run) for the GPT-2 PPL numbers (17.4 / 18.0 / 18.4), the ResNet-50 / ImageNet curves in Fig. 5, and the Table 1 finetuning entries, addressing No seed variance for headline claims and Table 1 gains are within noise.
• Release the missing logistic-regression code. Ship a minimal SGD/SVRG/VSGD/VSGD-PoCo/IVON/IVON-PoCo/L-BFGS pipeline for MNIST/Covertype/CIFAR-10 matching the Sec. 4.1 hyperparameters, resolving the Headline logistic-regression results are un-runnable gap in Reproducibility & code.
• Release an $\alpha$-SVRG reference implementation. Include the $\alpha$-SVRG optimizer used for Figs. 6, 7, 8b, and 9 (or pin the exact external repo/commit), addressing $\alpha$-SVRG baseline is not implemented.
• Add a convergence discussion. Either provide a rate analysis for VSGD-PoCo / IVON-PoCo under isotropic or diagonal $q$, or explicitly state that no rate is claimed and discuss where Thm. 2 does and does not transfer Johnson–Zhang guarantees, addressing No convergence theory.
• Recalibrate deep-learning framing. Soften "accelerates," "boosts," and "scales" in the abstract and Sec. 4.2 to reflect that Fig. 4b shows no matched-compute improvement over IVON $3\times$-bsz on GPT-2, and that ImageNet gains vanish when counted in data examples seen, addressing No matched-compute deep-learning speed-up.

Suggested

• Fix the Hessian-correction novelty claim. Rewrite the paragraph starting "To our knowledge no other Newton variant of SVRG implements Hessian corrections…" to acknowledge SVRG2 (Gower et al., 2018) and Chayti et al. (2024) as prior Hessian-correction work and state precisely how SVRH differs (expected outer Hessian averaged over $q_{\text{out}}$), per Hessian-correction novelty is overstated.
• Clean up Alg. 4. Rename h to h_in (or vice versa) consistently on the initialization line; describe $\rho_1, \rho_2$ as "outer-loop momentum coefficients" (plural); fix the "line 17" cross-reference (or replace with "we set $\alpha = 0.01$"); un-collapse the dense lines 7–10, per Algorithm 4 has notation drift.
• Reword the Thm. 2 framing. State in the abstract and Sec. 3 that SVRG is recovered as the deterministic ($\varepsilon \to 0$) limit of VSGD-PoCo rather than as a "special case," and note that sampling adds no measurable benefit in the isotropic case (Fig. 3), addressing "Special case" framing for Thm. 2.
• Ship missing configs for shipped experiments. Add an AdamW config and an IVON-$3\times$-bsz config for OpenWebText / GPT-2, and align the CIFAR-10 default script (or add reproduce_fig6.sh) with App. F.1 hyperparameters, so that a reader running the shipped code out of the box reproduces Fig. 6 and can populate Fig. 4a/4b's missing curves. Addresses CIFAR-10 ResNet-20 partially reproduces and GPT-2 OpenWebText run … baselines are also missing.
• Reconcile Fig. 4 caption naming. Attribute the 17.4 PPL to IVON-PoCoMo consistently (or explicitly note that momentum is off in this run), per Fig. 4 caption attributes an IVON-PoCoMo result to "IVON-PoCo".
• Add sensitivity ablation for $\rho_1, \rho_2$. Report how sensitive IVON-PoCoMo's final PPL / accuracy is to the outer-momentum coefficients on at least one of the paper's benchmarks, per New hyperparameters have no sensitivity study.
• Confirm the $\alpha$-schedule recommendation at scale. Repeat one of the schedule comparisons at the 125M GPT-2 / OpenWebText scale (or on ImageNet) before recommending increasing $\alpha$ against Yin et al.'s decreasing $\alpha$, per Ablation scope is narrow.
• Add significance testing to Table 1. Report paired-difference statistics (or at least standard deviations across seeds) for the ViT/Qwen/Llama finetuning gaps, per Table 1 gains are within noise.
• Ship finetuning, continual-pretraining, and wikitext-ablation code. At minimum one runnable pipeline per experiment (LoRA / OpenCLIP / Qwen / FineWeb / wikitext103), so App. E.2–E.4 and Table 1 become reproducible from the release. Addresses Finetuning pipelines (Table 1) are absent and Wikitext103 ablations and continual pretraining are absent.
• Qualify the "Code is available publicly on GitHub" statement. Either release the missing pipelines listed above or explicitly note which experiments the release does not currently cover.
• Broaden the convex baseline suite. Add at least one non-logistic-regression convex problem (ridge, SVM, or matrix completion) — especially for VON-PoCo — per Narrow convex baseline suite.