Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic Methods

Summary

The paper reframes action smoothness in continuous-control actor–critic learning as a property of the critic's geometry rather than the actor's output. Using implicit differentiation, the authors derive the local Jacobian of the greedy policy as $J_\pi = -(\nabla^2_{aa} Q)^{-1} \nabla^2_{sa} Q$ and, under strict concavity of $\nabla^2_{aa} Q$, obtain a spectral bound $L \le M/\mu$ on the induced policy's Lipschitz constant. From this factorization they propose PAVE, three critic-side losses that (i) suppress the mixed-partial norm $M$ via finite-difference MPR, (ii) align consecutive Q-gradients via VFC, and (iii) preserve action-Hessian concavity via a Hutchinson-trace Curv penalty. On six Gymnasium environments under both TD3 and SAC, PAVE achieves the best or competitive smoothness with no change to the actor, and appendices back the mechanism with $M_\mathrm{sup}$ reductions, strict-concavity rate improvements, cosine-flip drops, temporal-checkpoint evolutions, a full-factorial ablation, and a CAPS+PAVE ≈ PAVE comparison that supports critic-side sufficiency. The conceptual pivot is genuinely novel relative to actor-side Lipschitz regularizers (CAPS/GRAD/ASAP/LipsNet) and internally coherent.

The main conceptual limitation is that the "bound" is a conditional local identity: it requires an interior strict maximum with negative-definite action Hessian, a condition the paper itself reports as violated in 53–86% of visited states under base critics. So the language of "we prove that policy sensitivity is bounded by the ratio" in the abstract and "rigorously prove" in the contributions overstates a result the body correctly labels as "regularization motivation." A second gap is that the proxy losses target Frobenius-norm and trace surrogates for the spectral / max-eigenvalue quantities that appear in the bound; on Ant, Walker, and Hopper the VFC proxy captures only 3–31% of its intended target, so the alignment between proxy and theory is empirical rather than principled in the high-dimensional regime that most motivates the paper. Finally, the mechanism-level evidence (M_sup, µ, cosine flips, checkpoint drift) is TD3-only, no code is released, and several critical protocol details (FFT smoothness formula, base training hyperparameters, σ_base, dominant-axis selection) are deferred to citations or left implicit.

Strengths

• Conceptual contribution. Reframing action smoothness as a property of the critic's differential geometry is a genuinely different move from actor-side Lipschitz regularization, and the IFT-based factorization $J_\pi = -(\nabla^2_{aa} Q)^{-1} \nabla^2_{sa} Q$ is a clean way to make that move concrete.
• Mechanism-level empirical validation. The paper does not stop at reporting the composite smoothness score. It measures the exact analytic mixed-partial spectral norm M_sup, the strict-concavity rate of $\nabla^2_{aa} Q$, and the temporal cosine-flip rate of $\nabla_a Q$ — the three quantities the theory identifies — and shows that PAVE moves each in the predicted direction.
• Honest handling of failure modes and negative results. Remark 4.4 openly reports that the strict-concavity assumption often fails, Table 4 correctly flags Reacher's M_sup reduction as '—', Appendix J shows an SN-Critic collapse, and Appendix F states VFC proxy ratios as low as 0.03. These are all uncomfortable numbers that a less-scrupulous paper would hide.
• Ablations that support the specific mechanism. The full 2³ factorial (Appendix L) and the incremental ablation (Table 7) show that MPR+VFC without L_Curv actively worsens smoothness on Walker (2.095 > Base 1.828), which is exactly what the $M/\mu$ story predicts. CAPS+PAVE ≈ PAVE (Appendix M) is a genuinely informative test of critic-side sufficiency.
• Complementary critic-side baselines. The comparison against SN-Critic and GP-Critic in Appendix J directly answers the natural question "does the effect come from any critic-side regularization?" — a much stronger positioning than only comparing to actor-side methods.

Weaknesses

• Overclaim of theoretical guarantee. The abstract's "we prove that the sensitivity ... is bounded by the ratio" and the introduction's "rigorously prove that policy sensitivity is fundamentally governed by ..." present the bound as a general theorem, while Remarks 4.2 and 4.4 acknowledge that it is a conditional local identity holding at only 14–47% of visited states under unconstrained critics. This gap between the front matter and the body is the paper's most visible framing issue.
• Curvature preservation controls the wrong quantity for the bound. The bound depends on $\mu = -\lambda_\mathrm{max}$, i.e. the maximum eigenvalue, but L_Curv bounds the trace via Hutchinson. A negative trace does not imply a negative-definite Hessian; the eigenvalue-vs-trace gap is exactly the non-trivial step. The paper acknowledges this in Section 5.3, but the Neg.Def rates falling short of 1.0 in Ant/Hopper/Walker (0.64–0.86) are consistent with trace control being an imperfect surrogate that the paper does not analyze further.
• Proxy losses target Frobenius/trace surrogates for spectral/max-eigenvalue quantities. The MPR formal justification uses ‖A‖_2 ≤ ‖A‖_F to bound the theoretical M; this ensures the proxy is an upper bound minimizer but does not upper-bound the gap between the two. In wide, anisotropic Hessians (e.g. Ant, Humanoid), Frobenius-driven optimization may leave the spectral norm high even as the proxy decreases.
• VFC's link to the theoretical objective is empirically weak on high-dim environments. The proxy ratios in Table 13 are 0.03 (Ant), 0.10 (Lunar), 0.30 (Hopper), and 0.31 (Walker). The residual is dominated by higher-order and dynamics-driven contributions that need not be aligned with ∇²_sa Q · Δs, so on the very environments where PAVE's advantage is largest, VFC may be optimizing something other than the theoretical quantity. The factorial ablation combines VFC with other losses, so its individual contribution to reducing $M$ along dynamics-relevant directions is not isolated.
• Mechanism-level diagnostics are TD3-only. M_sup, negative-definiteness rate, cosine flips, checkpoint evolutions, factorial ablation, and critic-side baselines are all reported under TD3. SAC's stochastic actor and entropy regularization may interact differently with the critic geometry, so the mechanism claim as it applies to SAC currently rests on the outcome-level Table 2 alone.
• Table 5 caption and 14–47% range conflict with the table's own numbers. Table 5's "simultaneously suppresses M_sup and lifts the strict-concavity rate" is violated by the Reacher row (M_sup 16 → 26, Neg.Def 1.00 → 0.99), and the widely quoted "14–47%" base range silently excludes Reacher's 100%. Similarly, Appendix H reports "64–81%" whereas the main text and Table 15 imply "64–100%." These inconsistencies are all fixable, but they weaken the internal rigor of an otherwise diligent empirical section.
• Symbol overloading. δ denotes both the curvature margin (Alg. 1) and the observation-noise vector (Sec. 6.7); σ denotes the MPR perturbation scale, the CAPS perturbation scale, and the observation-noise magnitude bound. A careful reader tracking either symbol will have to disambiguate by context.
• Humanoid claim is a single Base-vs-PAVE pair. The "PAVE enabled TD3 to learn a non-trivial policy" claim (9.4× return) is presented without any actor-side or critic-side baseline comparison and without a component ablation, so it is not clear the effect is attributable specifically to the geometric losses rather than to the SiLU-unified critic or the extra regularization pressure in general.
• Cross-environment sm comparisons. The Humanoid discussion argues 0.76 is "competitive" with Hopper 0.95 and Walker 1.27, but sm depends on action dimensionality (17 vs 3 vs 6) and scale, so cross-environment absolute values are not directly comparable.
• Tight rankings inside standard deviations. Several bolded "best" cells (e.g. Reacher SAC PAVE 0.052 vs GRAD 0.046 with ±0.014 std; Ant SAC PAVE re 5706 vs GRAD 5729 with ±604) fall within one standard deviation, and Table 1 also has this pattern. The paper's high-level conclusion — that PAVE is a competitive, sometimes best, method — is robust to this, but the per-cell bolding suggests a sharper ordering than five seeds support.
• ASAP Hopper M_sup = 27627 outlier goes unremarked. If real, this is arguably the paper's most striking piece of evidence for actor-side regularization destabilizing the critic geometry. Leaving it uncommented invites the reader to suspect a measurement artifact.

Reproducibility & code

The instructions treat this as paper-code mode (code READ but not executed). In practice the released materials are effectively absent — the accompanying codebase directory is empty, and the paper never states a code URL. The reproducibility story therefore relies entirely on the methodology in the manuscript, which is well-organized but has several load-bearing omissions.

• No source code accompanies the paper. With no reference implementation, all numeric cells must be reproduced from the described methodology alone.
• Base training hyperparameters are absent. Learning rates, batch sizes, hidden widths, γ, τ, exploration noise, buffer size and target-entropy schedule are not stated for either TD3 or SAC. Tables 10–11 only give method-specific λ / σ, so a reproducer has to inherit SB3 defaults or match Kwak & Hwang (2026) exactly — the paper does not disambiguate.
• The FFT smoothness metric formula is deferred to citations. Because "PAVE achieves best smoothness" is the paper's headline claim, the exact bin weighting, per-dimension aggregation and normalization used in sm should be restated so numeric comparisons are unambiguous.
• σ_base is undefined. The scale-aware noise protocol δ ~ U(−σ,σ) ⊙ σ_base uses σ_base without stating how it is computed (per-dimension observation std over what distribution?). The Walker noise-invariance result cannot be reproduced without this.
• Dominant-axis and reference-state selection for the Q-gradient visualization is underspecified. "Maximum mixed-partial magnitude" is only defined once an aggregation over (s,a) is chosen, and the per-environment resulting (i,j) pairs and reference states are not tabulated.
• Humanoid-v5 hyperparameters are missing from Tables 10–11. Since the Humanoid experiment is used to argue that PAVE "enabled" learning, its λ's, training length and seed count are load-bearing.
• SN-Critic and GP-Critic configurations are not stated. Which critic layers are spectrally normalized, the singular-value bound, and whether GP is ‖∇_a Q‖² or ‖∇_(s,a) Q‖² are all inferred by the reader. The strong "SN-Critic collapses" conclusion is only as robust as the SN configuration choice.
• Coordinate-search protocol is under-described. The candidate grid, stopping criterion, and — most importantly — the selection metric (return? smoothness? joint?) for PAVE's λ's are not stated, even though the reported per-env values vary by 200×.
• Hutchinson trace estimator uses one probe with unquantified variance. Algorithm 1 samples a single Rademacher vector per state. In high-dim action spaces this is a noisy estimator whose bias/variance profile is not discussed.
• Sensitivity sweep uses different seeds and hardware and does not match Table 1. Table 21's ×1 Walker sm (1.52) differs by >20% from Table 1's PAVE Walker sm (1.272). The disclosure is honest but means the "λ₃ most sensitive" conclusion has not been shown to hold on the main-experiment regime.
• No standard deviations for several critical tables. Tables 14, 15, 20, 23 report point estimates without variability, making tight rankings (e.g. Reacher Base 16 vs PAVE 26) difficult to trust.

Recommended Changes

Essential

• Qualify the theoretical statements in the abstract and introduction. Rewrite "we prove that the sensitivity ... is bounded by the ratio" and "rigorously prove that policy sensitivity is fundamentally governed by ..." to reflect the conditional nature of the bound — the assumption of interior strict maxima and negative-definite action Hessian, and the paper's own report that the assumption holds in only 14–47% of states.
• Fix the Table 5 caption and the '14–47%' range. State "in most environments" (or explicitly note Reacher as an exception) in the Table 5 caption, and either report the full Base range (14–100%) or explicitly exclude Reacher, so the number a reader carries away matches the table's own contents.
• Add SAC mechanism-level diagnostics. Extend the M_sup / negative-definiteness / cosine-flip diagnostics to at least one SAC environment where PAVE claims a smoothness win, so the theoretical mechanism can be considered validated for stochastic actors as well as TD3.
• Add an isolated VFC diagnostic on high-dim envs. In light of the 0.03 (Ant), 0.10 (Lunar), 0.30 (Hopper), 0.31 (Walker) proxy-to-target ratios, either show that removing VFC from PAVE on these environments meaningfully hurts M_sup reduction along dynamics directions, or acknowledge in Section 5.2 that VFC's link to the theoretical objective in high-dim environments is empirical rather than principled.
• Rework the Curvature Preservation justification. Explicitly argue (or acknowledge as an empirical surrogate) the gap between trace control (what L_Curv bounds) and max-eigenvalue control (what the L ≤ M/µ bound requires), since the two are not equivalent and the paper currently glosses over the step.
• Soften the Table 7 ablation claim. Rewrite "directly validated our theoretical bound L ≤ M/µ by proving that minimizing volatility alone was insufficient if curvature vanished" to "is consistent with," and either add µ measurements for the intermediate +MPR+VFC configuration or reference the Sec. 6.5 µ measurements as the actual evidence.
• Release code and complete the hyperparameter specification. Publish an implementation and include the base TD3 / SAC training hyperparameters (learning rate, batch size, hidden dim, γ, τ, exploration noise, buffer size, target-entropy schedule), the exact FFT smoothness metric formula, σ_base's computation, and Humanoid's PAVE hyperparameters in the paper itself so the results can be reproduced without inferring SB3 defaults.
• Justify the Humanoid causal claim. Either add CAPS / GRAD / ASAP and per-loss component ablations on Humanoid-v5, or soften "PAVE enabled TD3 to learn a non-trivial policy" to "improved," so the 9.4× ratio is not carrying an unfalsified attribution.

Suggested

• Resolve symbol overloading. Use a distinct symbol for the observation-noise vector in Sec. 6.7 (not δ), and rename either the MPR perturbation scale or the observation-noise magnitude bound so σ is not simultaneously three different things.
• Explain the ASAP Hopper M_sup = 27627 outlier. Either investigate whether it is a genuine failure mode of policy-side regularization on the critic (which would strengthen the thesis) or verify the measurement.
• Report standard deviations in Tables 14, 15, 20, 23. Several tight rankings (e.g. Reacher Base 16 vs PAVE 26) are within plausible seed noise; adding stds would let readers weight the conclusions appropriately.
• Specify the dominant-axis / reference-state choices for the Q-gradient visualization. Add a table listing the chosen (state-dim, action-dim) pair and reference state per environment so Figures 1–3 are reproducible.
• Specify the SN-Critic and GP-Critic configurations used in Table 19. State which critic layers are spectrally normalized, the singular-value bound, and the exact GP form (‖∇_a Q‖² vs ‖∇_(s,a) Q‖², centered or one-sided) together with the reference λ.
• Relabel Eq. 14 as a directional first-order approximation. In Appendix F, drop "exact analytic" language for VFC so readers do not conflate proxy-vs-approximation with proxy-vs-ground-truth validation.
• Add a same-seed same-hardware reference point to the sensitivity sweep. So the "λ₃ most sensitive" conclusion can be tied to the main-experiment regime.
• Caveat cross-environment sm comparisons. In the Humanoid discussion, either compare against a within-environment reference (e.g. Base at similar return level) or note that raw sm values across 3-/6-/17-dim action spaces are not directly comparable.
• Reconcile the Appendix H "64–81%" figure with the main-text 64–100%. State clearly which subset of environments and which training stage the 64–81% refers to, or use consistent numbers throughout.