On the Convergence Rate of LoRA Gradient Descent

Summary

The paper studies gradient descent on the LoRA parametrization $W = W_0 + BA$ and asks how fast it converges when the original loss $L$ is Lipschitz smooth and lower bounded, but the reparametrized loss is not. Prior work either studies infinite-width limits, restricts to one-adapter-at-a-time variants (RAC-LoRA, Bernoulli-LoRA), or explicitly assumes bounded adapter norms — which effectively re-imposes Lipschitz smoothness. The conceptual move is to stack $A^T$ and $B$ into a single tall matrix $V$ so that the reparametrized loss becomes $L(E_1 V V^T E_2)$, then derive a "Lipschitz-like" descent lemma with extra higher-order terms in $V$. Because $\nabla J(V)$ carries a $V$ factor, a state-dependent learning rate $\eta_t \propto 1/(\|V_t\|^2 \|\nabla L\|)$ is needed to guarantee per-step descent; the price is a recursive coupling between $\eta_t$ and $\|V_t\|$. The authors control this by showing $\|V_t\|^2 = O(t)$ in the worst case, giving $\sum \eta_t = \Omega(\log T)$ and the headline $O(1/\log T)$ bound on the minimum gradient norm — recovering the classical $O(1/T)$ rate when adapters happen to stay bounded. The theoretical picture is complemented by CIFAR-10 logistic-regression and ResNet-18 experiments plus a TinyLlama fine-tuning study, all designed to exhibit the interplay between $\|V_t\|$, $\eta_t$, and observed convergence, and to motivate practical adaptive/normalized schedules ($\eta^{\mathrm{adapt}}$, $\eta^{\mathrm{adapt}_2}$, $\eta^{\mathrm{norm}}$). Overall this is a careful, well-motivated contribution to LoRA theory that fills a real gap. The main conceptual limits are that the guarantee is a best-iterate upper bound with no matching lower bound (so it is unclear whether the $1/\log T$ factor is intrinsic or a proof artifact), that the theoretically-cleanest schedule is impractical at scale, that the deterministic theory is validated only with minibatch SGD, and that several supporting empirical claims — most notably the large-init LLM regime — rest on hand-wavy reasoning and single-seed runs. An execution of the released code confirms the core empirical phenomena but also shows the headline adaptive schedule was silently broken as shipped.

Strengths

• Conceptual contribution. The stacking $V = [B; A^T]$ turns the two-matrix LoRA update into gradient descent on $J(V) = L(E_1 V V^T E_2)$ and cleanly connects the analysis to symmetric Burer–Monteiro. This is a genuinely useful reframing that survives the multi-matrix extension in Section 3.
• First non-asymptotic rate without a bounded-adapter assumption. Prior analyses either enforce bounded $A, B$ or work only asymptotically; establishing $O(1/\log T)$ under just Lipschitz smoothness and lower-boundedness of the original loss is a real gap-closing result relative to Ghiasvand et al. and Jiang et al.
• Honest calibration of the rate. The paper cleanly distinguishes the general $O(1/\log T)$ rate from the bounded-iterate $O(1/T)$ recovery, and explains the mechanism (the recursive $\eta_t$–$\|V_t\|$ coupling and the harmonic-series lower bound on $\sum \eta_t$) rather than hiding it in the appendix.
• Position-dependency insight. The observation that the LoRA reparametrization creates a stationary point at the origin — regardless of the original loss — is neatly extracted from the theory and offers a concrete mechanism for the "LoRA and full-rank fine-tuning may not converge to comparable solutions" observation of Shuttleworth et al.
• Core empirical phenomena reproduce. On execution, the two mechanisms behind the theory show up cleanly: for logistic regression the loss converges early while $\|V_t\|$ grows without leveling off ($\sim 1.15 \to 11.18$ over 1000 epochs), and for ResNet-18 the loss reaches interpolation and $\|V_t\|$ plateaus once it does. The vision-task schedule comparisons also reproduce (fast early descent, stable tail; $\eta^{\mathrm{norm}}$ best on ResNet-18).
• Careful scope statements. The LLM section correctly notes that in the large-$\|V_t\|$ regime the adaptive advantage collapses, and the candid admission that $\eta^{\mathrm{adapt}}$ is not tractable in standard LoRA implementations is welcome.

Weaknesses

• Rank-independence claim conflates the rate with hidden constants. Section 3.3 asserts as a "rigorous observation" that the rate does not depend on rank, but Theorem 3.5's $O(\cdot)$ hides constants involving $V_0$ and $J(V_0)$, both of which depend on $r$ through the stacked-adapter dimension. Only the $T$-dependence is rank-free — the same distinction the paper uses to criticize Jiang et al.
• No lower bound / tightness argument. The paper never argues $O(1/\log T)$ is the actual rate — only an upper bound achieved with the state-dependent $\eta_t$. It is possible the true rate is $O(1/T)$ and the $1/\log T$ factor is a proof artifact. The Section 4 assertion that "the asymptotic behavior is slower" than $O(1/T)$ overstates what a one-sided upper bound establishes.
• Deterministic theory, stochastic experiments. The rate and the schedule (8) are full-batch GD results, but every experiment uses minibatch SGD (the paper itself lists stochastic GD as unsolved future work). The experiments are best read as motivating, not validating, the theory.
• Best-iterate rather than last-iterate guarantee. Theorem 3.5 controls $\min_t \|\nabla J(V_t)\|^2$; the last iterate, which practitioners deploy, is not covered. Standard for nonconvex analysis but worth an explicit mention.
• Theory-cleanest schedule is impractical at scale. $\eta^{\mathrm{adapt}}$ requires $\|\nabla L(E_1 V_t V_t^T E_2)\|$, which materializes the reconstructed weight $BA$ and is not what standard LoRA computes — hence its omission from the ResNet-18 and LLM runs. The Section 3 exposition should state this up front and clarify how $\eta^{\mathrm{adapt}_2}$ approximates rule (8) beyond the pointer to Eq. (23).
• Comparison "in the same range of values" is loose. The reproduced logistic-regression runs show the nonconstant schedules peaking at effective rates of $\sim 0.22$–$0.27$, several times above the constant grid $[0.03, 0.05, 0.07]$ they are compared against. The observed benefit is partly a high-early/low-late schedule shape no tested constant can match, which should be stated as such.
• BatchNorm-off caveat is buried in the appendix. ResNet-18 runs use model.eval() (BatchNorm frozen), a substantive departure from standard fine-tuning that changes the loss surface on which the step-size-modulating schedules are assessed. A BatchNorm-active control run, and lifting the caveat into Section 4, would help.
• LLM large-init explanation is hand-waved. The rationale for the collapse at $\sigma = 1/r$ ("$\|V_t\|$ very large but grows extremely slowly relative to its size") is asserted, not measured — and the reproduction (below) complicates it.
• LLM comparison lacks published tuning details. The central LLM claim depends on "constant learning rates of similar magnitudes," but the paper never states the $\alpha$ values, the constant baselines, the number of steps, or the training slice for TinyLlama. Table 1 covers only CIFAR-10.
• Empirical scope and variance. All results are single-seed at a few hyperparameter points on one image classifier and one small chat LLM. Ordering claims like "$\eta^{\mathrm{norm}}$ outperforms $\eta^{\mathrm{adapt}_2}$" rest on tail-loss gaps ($\sim 0.003$) well within plausible seed noise.
• "Almost no impact" of $E_1, E_2$ overstates. The proof is insensitive to which block of $V V^T$ is extracted, but the block determines the representable update class (top-left gives symmetric PSD $B B^T$, not arbitrary $BA$). The Discussion suggests unwarranted generality.
• Notation slips. (i) "$L$ is not Lipschitz smooth in $B$ or $A$" reuses $L$, defined over $W$, for the composition $L(BA)$; (ii) the stated stacking order "$A^T$, $B$" contradicts the extractor definitions that need $B$ on top; (iii) $A_2$ is typed $\mathbb{R}^{r \times n_1}$ where $\mathbb{R}^{r \times n_2}$ is needed; (iv) $L \geq 1$ is imposed without comment. None are load-bearing but together they slow a careful reading. The load-bearing linear-in-$t$ coefficient in the $D_t$ bound is also not spelled out.

Reproducibility & code

Execution used the shipped training paths (patched where necessary); figures and one experiment branch had to be reconstructed. Overall 3/3 headline claims and 9/12 total reproduced (score $\approx 0.89$).

• Headline adaptive schedule was broken as shipped. The eta_adapt branch in main.py was a stray if instead of elif, so an adaptive run fell through to the constant path ($\text{lr}=1.0$) and diverged to NaN from epoch 1; two further bugs (unassigned best_step, uncomputed lipschitz_L) crashed the save step. After a control-flow fix the run converged (final loss $\sim 0.44$) and the claim reproduces, but the paper's adaptive figures cannot be regenerated from the release unedited.
• Loss/norm phenomena reproduce. The logistic-regression $\|V_t\|$-unbounded-with-converged-loss fingerprint and the ResNet-18 plateau-at-interpolation behavior both reproduced directly from the checkpoint histories, as did the vision schedule comparisons ($\eta^{\mathrm{norm}}$ lowest/steadiest, $\eta^{\mathrm{adapt}_2}$ next).
• Large-init LLM branch is missing and behaves differently than described. llm.py hard-codes the $A$-init scale at $1\mathrm{e}{-3}$ with no $\sigma = 1/r$ branch. When run, the large-init adaptive step size did become nearly constant ($\sim 3.5\mathrm{e}{-5}$) — but $\sim 500\times$ below $0.02$, so the model essentially did not train (loss flat near $2.0$). The paper conveys "behaves like a constant," not "did not learn."
• Small-init LLM claim only partially holds. With $\sigma = 1\mathrm{e}{-3}$, the normalized schedule clearly beat the constant run (final MA-10 loss $\sim 1.40$ vs $\sim 1.58$), but the adaptive schedule tied the constant run with marginally higher step-to-step variance, so "faster and more stable" holds for $\eta^{\mathrm{norm}}$ and not clearly for $\eta^{\mathrm{adapt}_2}$.
• LoRA-vs-full-rank: only the logistic-regression arm verified. At matched $\text{lr}=0.05$, LoRA was clearly slower ($\sim 0.44$ vs full-rank $\sim 0.34$), matching Figure 7. No matched full-rank ResNet-18 baseline was runnable, so Figure 8 was not verified; neither shell script runs a no-LoRA sweep.
• No plotting code; missing/mismatched drivers. The repo ships only a learning-rate-finder notebook, so all figures were reconstructed from captions. The 1000-epoch Figure 4 sweep is not scripted (only --epochs 60); resnet_script.sh iterates $\{0.03, 0.04, 0.05\}$ rather than Table 1's $\{0.03, 0.04, 0.045\}$; and llm.py logs step size but not gradient norm, so Figure 5's gradient-norm panels are not reproducible as published.
• Single seed throughout. Scripts hard-code --seed 42; no variance information supports the comparative claims.
• Possible ResNet-18 construction bug. In models.py:get_cifar_resnet18 the CIFAR head replacements (conv1, maxpool, fc) appear to be applied after peft.get_peft_model, so they may not propagate into the trained model. The runs still converged sensibly, so any effect is not obviously fatal, but the path should be verified.

Recommended Changes

Essential

• Qualify the rank-independence claim. In Section 3.3, state that only the $T$-dependence is rank-free while the hidden constants (via $V_0$, $J(V_0)$) generally depend on $r$, mirroring the paper's own critique of Jiang et al.
• Address tightness of $O(1/\log T)$. State in the theorem or conclusion that no matching lower bound is established; provide a concrete example (or heuristic) where the $1/\log T$ factor is achieved, or flag as open whether the true rate is faster, and soften the "asymptotic behavior is slower" phrasing in Section 4.
• Clarify the GD-vs-SGD gap. State that Theorem 3.5 and schedule (8) are full-batch GD results while all experiments are minibatch SGD, and frame the experiments as motivating rather than validating the deterministic rate.
• Fix and release the broken adaptive code path. Correct the if/elif control flow (and the best_step/lipschitz_L errors) in main.py so the released eta_adapt schedule runs end-to-end, and confirm the published adaptive figures come from the fixed code.
• Report LLM tuning details and reconcile the large-init story. Specify the $\alpha$ values, constant baselines, step count, and training slice for TinyLlama; add a direct $\|V_t\|$ trajectory at $\sigma = 1/r$; and note that the reproduced large-init adaptive run collapsed to a near-zero step size and failed to learn, rather than merely tracking a constant.
• Release the missing code paths and plotting. Publish plotting code for all Section 4 / Appendix B figures, a driver for the 1000-epoch Figure 4 sweep, a $\sigma = 1/r$ branch in llm.py, gradient-norm logging for the LLM runs, the missing $0.045$ constant in resnet_script.sh, and a no-LoRA driver matching Figures 7–8.
• Verify the ResNet-18 model construction. Confirm (and reorder if needed) that the CIFAR head replacements in get_cifar_resnet18 propagate into the PEFT-wrapped model, and rerun Figure 2 if the shipped code differs from what produced it.
• Fix the notation slips. Rephrase "$L$ not Lipschitz smooth in $B$ or $A$" in terms of $L(BA)$/$J$; correct the stacking order to $[B; A^T]$; type $A_2$ as $\mathbb{R}^{r \times n_2}$; note $L \geq 1$ is WLOG; and spell out the $1/(4\sqrt{2}L)$ coefficient in the $D_t$ linear bound.

Suggested

• State the best-iterate limitation. Add one sentence to Theorem 3.5 clarifying it controls $\min_t \|\nabla J(V_t)\|^2$, not the last iterate.
• Report the effective-rate range. In Section 4, note that the adaptive/normalized schedules reach effective rates well above the constant grid early in training, so the "same range of values" comparison reflects a schedule shape rather than a matched magnitude.
• Add a BatchNorm-active control and cross-seed variance. Include at least one ResNet-18 run without model.eval(), lift the caveat into Section 4, and re-run the CIFAR-10 experiments over 3–5 seeds with variance bands so ordering claims (e.g. $\eta^{\mathrm{norm}}$ vs $\eta^{\mathrm{adapt}_2}$) can be assessed against noise.
• Explain $\eta^{\mathrm{adapt}_2}$ and soften the $E_1, E_2$ remark. Briefly relate the loss-based surrogate $\eta^{\mathrm{adapt}_2}$ to Eq. (8) (and how $L$ is absorbed into $\alpha$), and reframe the extractor remark as "the proof carries over for any block extraction, though the representable update class depends on the block."
• Reconcile Figure 5's caption and prose about which adaptive schedules are actually run for the LLM.