Learning Human-Robot Collaboration via Heterogeneous-Agent Lyapunov Policy Optimization

Summary

The paper proposes HALO, a heterogeneous-agent Lyapunov policy optimization framework for human–robot collaboration (HRC) that reframes decentralized MARL updates as a dissipative dynamical system in policy-parameter space. The conceptual move is to define a rationality gap $V(\theta) = \tfrac{1}{2}\|\mathbf{u}_{\text{ind}} - \mathbf{u}_{\text{team}}\|^2$ between the concatenated independent policy-gradient field and the true team ascent direction, treat $V$ as a Lyapunov candidate, and project each decentralized update onto the half-space $\{\mathbf{d} : \nabla V^\top \mathbf{d} \leq -\sigma V\}$. KKT conditions yield a closed-form projection whose only extra ingredient is $\mathbf{h} = \nabla_\theta V$, computed as a transpose-Jacobian-vector product via double back-propagation — so the practical cost is one extra backward pass rather than an explicit Hessian. Theorems 5.2 and 5.3 give monotonic contraction of $V$ under a step-size condition and asymptotic convergence to a rationality-agreement manifold under Robbins–Monro schedules. Empirically, nine Isaac Lab scenarios grouped into OSP / SCT / SLH tasks show that HALO improves overall success rate (86.0% vs 82.1% for PCGrad, 80.8% for HATRPO, 77.1% for HAPPO), drives the gap $V$ down to 0.09 (vs 4.89 for HAPPO), and raises gradient alignment to 0.91; a Unitree G1 deployment illustrates coordinated transport under obstruction. The idea of certifying stability in the parameter-update space (rather than in the trajectory / constraint space of most Lyapunov-safe-RL work) is genuinely novel and cleanly formalized, and the closed-form projection is attractive because it slots into standard MARL trainers with minimal overhead. My main conceptual reservation is that the theoretical apparatus assumes independent per-agent parameters, whereas Appendix B.3 states the tactical coordinator is implemented as a shared-parameter HAPPO — the empirical numbers are then not measuring the setting the theory targets. Two central metrics (GCR, cos $\phi$) are used without formal definitions, several table renderings garble critical ablation and real-world evidence, and the "significantly faster convergence" line in Fig. 2 is contradicted by the Conv. step column of the paper's own Table 1.

Strengths

• Conceptual contribution. Elevating the rationality gap to a Lyapunov potential in policy-parameter space, and then rectifying decentralized updates via a minimum-norm projection onto the descent half-space, is a clean and reusable idea. It gives a formal handle on the rotational component that Balduzzi et al. identify in decentralized MARL, without requiring value-factorization structure or opponent modelling.
• Closed-form update. The KKT-derived projection $\mathbf{d}^* = \mathbf{u}_{\text{ind}} - \lambda^* \mathbf{h}$ with $\lambda^* = \max(0, \psi/(\|\mathbf{h}\|^2 + \epsilon))$ is analytic, involves a single scalar dual variable, and drops into any actor–critic pipeline. This is a real practical advantage over prior game-dynamics remedies that require Jacobian access or centralized modelling.
• Scalable HVP implementation. Computing $\mathbf{h} = \nabla_\theta V$ via double back-propagation rather than materializing the joint Jacobian keeps the per-step overhead modest for the MLP sizes used, and the memory-overhead claim (<21.1%) is at least directionally plausible for the [256, 256, 128] actor/critic.
• Multi-metric evaluation. Beyond success rate, the paper reports gradient alignment, the rationality gap itself, gradient conflict rate, convergence steps and final return — a broader battery than most MARL papers. Figure 4a/b visually confirms the qualitative story (monotonic $V$-decay and rising cos $\phi$) that the theory predicts.
• Real-world deployment. Moving from Isaac Lab to a Unitree G1 with MoCap is nontrivial, and the qualitative behaviours (adaptive squatting to keep the load horizontal; sustained coordination through 20 s partner obstructions) illustrate that the parameter-space stability translates into interpretable physical robustness.

Weaknesses

• Implementation contradicts the theoretical setting. Appendix B.3 states that the tactical coordinator uses a shared-parameter HAPPO, while the entire theoretical apparatus — the stacked joint parameter vector $\theta = [\theta_1^\top, \dots, \theta_N^\top]^\top$, the non-symmetric cross-Jacobian across agent blocks, and the rationality gap between decentralized and team gradients — is built for genuinely independent per-agent parameters. The paper explicitly says such sharing is "infeasible in heterogeneous HRC settings", yet appears to use it. Either the appendix wording is wrong or the empirical numbers are measuring something other than what the theorems govern.
• "Significantly faster convergence" is contradicted by the paper's own table. Figure 2b's caption asserts HALO converges "significantly faster", but Table 1's Conv. step column has HALO at 1.3B, tied with HATRPO and slower than PCGrad's 1.2B (which is the bolded best). The claim needs to be qualified or grounded in a different notion of convergence.
• Central formal expressions have avoidable errors. The non-symmetric-Jacobian condition is written with '=' instead of '$\neq$', giving the opposite of the intended claim; the local surrogate $J_i$ is defined via the raw $Q_{\text{tot}}$ in Sec. 4.1 but via the centralized advantage $\hat{A}_{\text{tot}}$ in Sec. 3.2; the KKT case split in Sec. 4.2 announces "two operational regimes" but the active-regime multiplier is only recoverable from Algorithm 1; and the Appendix A.1 chain rule drops the transpose exactly where non-symmetry makes it consequential. Individually each is small, but they cluster at the derivations that carry the paper's contribution.
• Theory–algorithm gap around step sizes. Theorem 5.2's monotonicity bound $\eta_k \leq 2\sigma V(\theta_k)/(L\|\mathbf{d}^*_k\|^2)$ forces $\eta_k$ to shrink at least as fast as $V(\theta_k) \to 0$, which is in tension with the Robbins–Monro hypothesis $\sum \eta_k = \infty$ that Theorem 5.3 relies on. The two theorems are presented as complementary but their step-size assumptions are not reconciled.
• Learning-proxy vs script claim is promised but not measured. The introduction says "later demonstrated empirically … that replacing static scripts with learning-capable humanoid proxies [is] a computational imperative", but the simulation study only compares MARL algorithms against one another; the only script comparison is a small n=5 real-world Table 3 vs Robot-Script (itself an IPPO baseline, not a true script). There is no matched-compute simulation ablation of "learning partner vs pre-scripted partner".
• Overclaimed real-world stability. The prose calls HALO's SLH-under-halting behaviour "exceptional stability", yet Table 3 lists HALO's drop rate as 20% — identical to PCGrad. The win is in drift and tilt rate, not in dropping the object.
• Small real-world sample. All real-world claims rest on n=5 trials per method-task cell. Drop-rate differences (0% / 20% / 60% / 100%) and the Wait boolean cannot support the confidence that the surrounding narrative places on them.
• Rendering issues obscure central evidence. The ablation table's 12-symbol block does not partition cleanly into rows consistent with the caption ("Static projection" ends up with P=×, contradicting its name); Table 3's SLH row cannot be parsed to tell whether Robot-Script completes the task; and the sentence "In OSP and SCT tasks, it significantly reduces time-to-destination (76.2 s)" attaches an SCT-only number to both tasks.
• Positioning/citation mismatches. The characterization that CTDE methods "regularize update magnitudes rather than geometric directions" overlooks that value-factorization methods (QMIX/QTRAN/QPLEX) constrain the value class rather than step magnitudes; Zhao et al. (2023), whose title is "Multi-agent first order constrained optimization in policy space", is cited as the authority for solenoidal / limit-cycle claims that Balduzzi et al. (2018) actually establish; and Yang & Nachum (2021) — "Representation matters: Offline pretraining for sequential decision making" — is attached to a Riemann–Finsler claim it does not make.
• Theory–practice residual not acknowledged. Theorem 5.3 concludes $V \to 0$, but the reported steady-state $V = 0.09$ is presented as confirmation of the theorem rather than as the finite-training / stochastic-gradient residual it is.

Reproducibility & code

The released materials referenced by the project page do not fully cover the specifications needed to reproduce the reported numbers; the following gaps are visible from the manuscript and the veritas assessment.

• Metric definitions. GCR — introduced as "the primary stability indicator" — is never mathematically defined; the gradient-alignment cos $\phi$ used in Table 1 and Fig. 4b/5c/6 also lacks an explicit formula. Different natural definitions (per-step vs seed-averaged; sign-disagreement fraction vs $\langle \mathbf{u}_{\text{ind}}, \mathbf{u}_{\text{team}} \rangle < 0$ episode fraction) give substantially different absolute values, so the 4.2% / 0.91 headline numbers cannot be interpreted or replicated without a formula.
• HALO-specific hyperparameters. Algorithm 1 requires $\eta$, $\sigma$, $\epsilon$, but Appendix Table 7 only records shared PPO knobs. The stability constant $\sigma$ (which controls the projection's contraction rate), the damping $\epsilon$, and the "adaptive scheduling ($\eta$)" and "alignment rectification (cos $\phi$)" components ablated in Table 2 are named but never given as values or equations.
• Task suite. The nine scenarios (Alignment / Turnaround / Corner entry / Narrow gate / S-shaped / U-shaped / Facing / Lateral shuffle / Pivoting) are described only in one-line prose. Reward functions, per-scenario success criteria, physical geometry (opening angle, gate width, board mass/friction) and the training-side human proxy are not specified; only the IPPO baseline's proxy (Table 8) is documented. This is the largest reproducibility gap: every headline success rate depends on these unreleased Isaac Lab task definitions.
• Evaluation protocol. The number of seeds behind Table 1's mean ± std, the aggregation across the nine scenarios, the operational definition of "convergence step" (return threshold, moving-average window), and the error-bar semantics in Figures 3–6 are not stated.
• Wall-clock and memory profile. The 13.1% wall-clock reduction and <21.1% memory-overhead claims depend on hardware, number of parallel Isaac Lab environments, and mini-batch size, none of which are given.
• Sim-to-real transfer. Despite the section title "sim-to-real transferability", no domain randomization schedule, low-level controller re-training scheme, or MoCap-to-policy latency handling is described. Table 3's numbers therefore cannot be reproduced from paper contents alone even if the trained checkpoints were shared.
• Real-world sample size. All real-world conclusions rest on n=5 trials per cell, well below what is needed to distinguish 20% from 60% drop rates with any confidence.

Recommended Changes

Essential

• Reconcile shared-parameter HAPPO with the heterogeneous framing. Either state that the tactical coordinator uses fully independent per-agent parameters (matching the theory) and correct Appendix B.3, or explicitly discuss how the shared-parameter implementation instantiates the joint parameter vector $\theta$ and how the non-symmetric cross-Jacobian argument still applies.
• Fix the central mathematical expressions. Replace '=' with '$\neq$' in the non-symmetric-Jacobian condition; pick either $Q_{\text{tot}}$ or $\hat{A}_{\text{tot}}$ as the definition of $J_i$ and use it consistently in Sec. 3.2, Sec. 4.1 and Algorithm 1; write out both active and inactive KKT regimes explicitly in Sec. 4.2; and add the transpose to the Appendix A.1 chain rule so ${\nabla_\theta V = J_e^\top e}$ is unambiguous.
• Define GCR and cos $\phi$ formally. Add the mathematical definition, the averaging protocol, and the estimator (mini-batch vs full-batch, per-step vs windowed) somewhere in Sec. 6.1 or the appendix.
• Document the task suite. Release reward functions, per-scenario success criteria and physical parameters for all nine OSP/SCT/SLH scenarios, and describe the training-side human proxy policy, observation and reward (not just the IPPO baseline's Table 8 proxy).
• Publish HALO hyperparameters. Add $\sigma$, $\epsilon$, the $\eta$-schedule form, and the "alignment rectification" component to Table 7 or a new table, and specify the number of seeds behind every mean ± std.
• Requalify the convergence and stability claims. Either qualify "significantly faster convergence" (e.g., "faster than HAPPO") or ground it in a different threshold-based Conv. step definition — the current Table 1 does not support it. Replace "exceptional stability" in SLH with a qualified claim that notes the matched 20% drop rate against PCGrad, and split the "76.2 s" statement so the OSP (61.7 s) and SCT (76.2 s) numbers are each attached to their own task.
• Reconcile the two convergence theorems. Explicitly discuss why the monotonicity bound $\eta_k \leq 2\sigma V(\theta_k)/(L\|\mathbf{d}^*_k\|^2)$ is compatible with $\sum \eta_k = \infty$, or state that Theorem 5.3 only requires $\liminf V = 0$ so per-step monotonicity is not needed.

Suggested

• Fix the ablation table rendering. Redraw Table 2 with a plain grid so each of P / $\eta$ / cos $\phi$ has its own column and each variant its own row, and add prose walking through which component contributes which fraction of the SR and gap improvements.
• Fix Table 3's SLH row. Separate the Wait column from the numeric columns and explain what each "–" denotes; report whether Robot-Script completes the halting task and, if not, mark it as a failure rather than an ambiguous dash.
• Add a learning-partner vs scripted-partner ablation in simulation. At matched compute, train HALO with a learning humanoid proxy and with a fixed scripted proxy and report the gap on OOD test scenarios. This is the comparison the introduction promises but does not deliver.
• Increase the real-world sample size. Raise n from 5 to 15–20 per method-task cell, or at least report per-trial values so drop-rate and drift comparisons can be interpreted.
• Repair the citations. Reattribute the solenoidal/limit-cycle claim in Sec. 3.3 / 4.1 (currently "(Zhao et al., 2023)") to a source that establishes it, and either correct the Yang & Nachum (2021) key or replace the descriptive clause about Riemann–Finsler metrics in Sec. 2.
• Acknowledge the finite-training residual. Add a sentence noting that the empirical $V \approx 0.09$ is the stochastic-gradient / finite-horizon residual and not itself a numerical demonstration of the $V \to 0$ theorem.
• Report hardware and rollout parallelism (GPU model, number of parallel Isaac Lab environments, batch size, framework) so the 13.1% wall-clock and <21.1% memory numbers are auditable.
• Document the sim-to-real pipeline (domain randomization schedule, low-level controller retraining, MoCap-to-policy latency) so Table 3 and Figure 8 can be interpreted as a repeatable experiment rather than a demonstration.