Chebyshev Policies and the Mountain Car Problem: Reinforcement Learning for Low-dimensional Control Tasks

Summary

This paper makes two entangled contributions to reinforcement learning on low-dimensional control tasks. First, it presents what appears to be the first closed-form analytical solution of the classical continuous Mountain Car problem, closing a 36-year gap. Using an energy-based spatial reformulation of the pendulum-like dynamics, a Cauchy–Schwarz argument yields the unconstrained optimum $\alpha = C\dot{x}$, which the authors then reintroduce constraints for to obtain a piecewise worst-case policy $\pi_{\text{ana}}$ with switching phase-1/phase-2 constants and a bootstrap term. Concretely, $\pi_{\text{ana}}$ reaches mean return 99.39 on the Gymnasium environment, against which the best deployed SB3-Zoo baselines (ARS 96.67, SAC 94.61, PPO 93.91) have residual regret 2.7–5.5 return points — enough to motivate the second contribution. Second, the paper introduces Chebyshev policies: multi-variate Chebyshev polynomial approximators used as drop-in replacements for MLP actors (and critics) in policy-gradient and random-search algorithms. The claim is that on Mountain Car, Chebyshev policies trained with ARS/PPO/REINFORCE reduce ARS's regret by a factor of 4.18 with roughly 277× fewer parameters, and that the improvement generalises qualitatively to Pendulum and to a real-world 1-DOF Quanser Aero 2 helicopter testbed, including sim-to-real transfer. The conceptual move — pushing dense-in-$C(S)$ polynomial bases into RL as an alternative to MLPs on tasks where the optimal policy is smooth and low-dimensional — is a genuine and welcome one, and the paper is unusual in framing the choice of policy class from a first-principles analytical starting point rather than from ablation. My main reservations concern (i) the treatment of the reference point for regret comparisons and the best-of-20-vs-deployed asymmetry, (ii) several under-derived analytical steps in the $\min_{x_0}\max_{\pi}$ argument and Lemma 2.3 proof, (iii) an outright reproducibility failure for CH-3-PPO on Mountain Car, where 13 of 20 retrained seeds fail to reach the goal, and (iv) release-quality gaps in the shipped code where several load-bearing experiments (PPO-MLP sweep, CH-3-SAC, discrete-time verification) have no runnable driver.

Strengths

• Conceptual contribution. Closing a 36-year gap with an actual closed-form worst-case policy on a canonical benchmark is a real contribution. The energy-based spatial reformulation, use of Königsberger's oscillator theorem, and Cauchy–Schwarz argument for the unconstrained optimum are elegant, and the reduction to $\alpha = C\dot{x}$ is a genuinely surprising insight.
• First-principles design of a new policy class. Motivating Chebyshev policies from the tameness of the analytical solution — orthonormality, universality via density in continuous policies, controllable smoothness through max-degree — is a cleaner design story than the usual post-hoc justification for RL architecture choices.
• Breadth of empirical support. The paper does not stop at Mountain Car: it evaluates Chebyshev policies on Pendulum-v1 and on a real-world Quanser Aero 2 testbed (both simulation and hardware), with three different algorithms (ARS, PPO, REINFORCE) plus a Chebyshev-representation-layer + SB3 variant (Table 6) that acts as a useful sanity check.
• Practical engineering relevance. The observation that PPO on the Aero 2 resorts to bang-bang control (with excess motor heat) while CH-3-PPO tracks smoothly is a compelling operational win, and the sim-to-real degradation numbers (2.15× for PPO vs 1.13× for CH-3-PPO) fit the paper's broader thesis.
• Rich appendix. Appendix A covers two-phase trajectories, wall interactions, the $\min_{x_0}\max_{\pi}$ reduction, and a discrete-time verification via Matlab fmincon. Appendix C reports architecture sweeps for PPO-MLP (Table 3) and CH-PPO degree (Table 5), and includes the negative result that REINFORCE fails to train any goal-reaching MLP policy (960 attempts), which strengthens the argument that the policy class matters here beyond optimiser choice.
• CH-3-ARS and CH-6-ARS results are robust. Independently retraining CH-3-ARS on Mountain Car (20 seeds, full 80k steps) gives a best-agent mean return of 98.95 against the paper's 98.74; retraining CH-6-ARS on Pendulum (8 seeds where the paper uses 30) gives grid mean −151.14 vs −150.80 with matching std, min, and max. Two of the three headline Chebyshev cells reproduce cleanly.

Weaknesses

• Reference point for the 4.18× headline. The regret-reduction factor is computed with respect to $\pi_{\text{ana}}$ (mean 99.39), not the trivial upper bound of 100 nor the per-$x_0$ optimum $\pi_{\text{opt},x_0}$ (mean 99.59). Recomputed against 100, the ARS-vs-CH-3-ARS factor is ~2.6. Neither the abstract nor Section 5 states which reference is used; the choice matters because CH-3-ARS still leaves residual regret ≈ 0.65 that later work will use as a baseline.
• Regret framing as "surprisingly high". A ~3–5 point gap on a 100-scale is comfortably above the RL-Zoo solved threshold, but calling it "surprisingly high" rests on the reader accepting $\pi_{\text{ana}}$ as the natural yardstick. A measured framing would still support the paper's motivation.
• Table 1 and Table 4 numerical inconsistency for CH-3-ARS. Reported mean $R = 98.74$ sits below the reported minimum $R_{\min} = 98.95$ over the same 100 evaluation points, which cannot both be true; the same values appear in both tables. Since CH-3-ARS is the headline policy, this needs to be checked and the 0.65 regret re-verified (retraining a fresh CH-3-ARS gives min 98.74 with mean 98.95 — the two values look swapped).
• Best-of-20 vs deployed-baseline asymmetry. Chebyshev policies are the best-mean-return out of 20 seed trainings, while the MLP baselines are single deployed SB3-Zoo agents. Extending Tables 1 and 4 to mean ± std over the 20 seeds — as Table 5 already does — would make the ~2-point improvement much more credible.
• Missing linear-policy ablation. The analytical optimum is essentially linear in $v$, yet Table 5 shows CH-PPO at $d=1$ collapses to return −0.1. Without a plain linear baseline (or a $d=1$ CH result for ARS and REINFORCE), the paper cannot separate "Chebyshev basis is a better representation" from "higher-degree polynomials happen to be easier to optimise".
• Under-derived worst-case starting position. The claim that the worst-case $x_0$ sits at the $U_g$ minimum $\hat x \approx -\pi/6$ is asserted from equation (5) but not derived; (5) is only the feasibility condition, and $\ell$ depends on the whole trajectory geometry. $\pi_{\text{ana}}$'s definition hangs on this step.
• Deployed $\pi_{\text{ana}}$ is not truly "independent of $x$". Theorem 2.4 gives an $x$-independent unconstrained optimum, but the deployed $\pi_{\text{ana}}$ switches $C$ between phase 1 (4.3346) and phase 2 (4.8358) as a function of $x$ and adds a state-dependent $\alpha_{\text{boot}}$. The 4.3346 constant, in particular, is stated as fixed but corresponds specifically to the worst-case $x_0$; earlier text says $C_{1,k}$ depends on $x_0$.
• Small proof-sheet defects around the main lemma. The Cauchy–Schwarz equality gloss reads "cosine being zero" where it should be "cosine being one"; the introductory sentence of the Lemma 2.3 proof is garbled ("g minimizing ‖f‖ …" with a bare $\langle f,g\rangle \le \lVert f\rVert$); the "reaches the goal fulfills (5) to strict inequality" phrase in the Theorem 2.4 proof is a run-on. None affect correctness, but this is the paper's main contribution and should read clean.
• Parameter-count claim (277×) is off by 62 in the intermediate breakdown. Footnote 4's arithmetic $2\cdot 65 + 64\cdot 65 + 65 = 4355$ undercounts the first Linear(2,64) layer (should be $3\cdot 64 = 192$, not $2\cdot 65 = 130$); the correct total is 4417. Interestingly, $4417/16 = 276.06$ matches the reported 277 much better than $4355/16 = 272.2$, so the ratio is right but the intermediate step is wrong. Please also state which Chebyshev components ($\mu$ only? $\mu + \sigma$? critic?) enter the denominator.
• Cross-environment evaluation protocol under-specified. Table 2's evaluation protocol is stated only for Pendulum (50×50 grid). MC and Aero 2 columns rely on the reader inferring the protocol from other sections. Given that Table 2 supports the abstract's cross-environment "consistently outperform" claim, this deserves a caption sentence.
• Sim-to-real power-consumption reversal on Aero 2. Tables 8/9 show CH-3-ARS and CH-3-PPO power drop 3–4× on the real system, while PPO's power rises from 37.05 W to 52.60 W despite similar voltages. The paper discusses pitch-deviation transfer but is silent on this reversal, which quietly undermines the sim reliability.
• Pendulum state dimension ambiguity. $S^1 \times [-1,1]$ is a 2-manifold, embedded in $\mathbb{R}^3$. The paper calls it "3-dimensional", but Chebyshev basis count $(d+1)^n$ differs by orders of magnitude between $n=2$ and $n=3$ at the paper's best-$d$ Pendulum results — the choice of $n$ used should be stated.
• Cross-referencing / labelling glitches. Table 5's title reads "CH-3-PPO" though the whole table sweeps degrees 1–50, self-contradicting the naming convention. Table 6 quietly uses CH-4-PPO alongside CH-3-ARS and CH-3-SAC while claiming the results "align" with the native CH-3-PPO of Table 4. The $x$ range for $\lVert\pi-\pi_{\text{ana}}\rVert_2$ is $[-1.2, 0.45]$ (goal at 0.45) but no note explains the asymmetry. Table 3's $[64,64]$ PPO mean (94.22) differs from Table 1's (93.91) purely because of a 20-vs-100 evaluation grid, yet "match the one reported on RL Baselines3 Zoo" invites confusion.
• $\sigma$-initialisation for Chebyshev policies is ambiguous. "Initialized to 1" can mean either $\theta^{(\sigma)}_{0,\ldots,0}=1$ with other $\sigma$-coefficients zero, or the constant coefficient set to 1 while higher $\sigma$-coefficients are drawn from $\pm 10^{-3}$. The natural reading conflicts with "all other polynomials … $\pm 10^{-3}$".
• REINFORCE-vs-MLP framing. "None solved … only one (RAdam[8]) at times reaches the goal with a mean return of about 60" is internally in tension — 60 is a partial success, though below the RL-Zoo threshold of 90. Naming that threshold inline would tighten the sentence.

Reproducibility & code

Executing the release confirms two Chebyshev headline numbers, produces an outright disagreement on a third, and exposes a set of release-quality issues (missing scripts, hardcoded selection rules, packaging defects) that make even the successful cases require substantial reverse-engineering.

• CH-3-PPO Mountain Car did not reproduce. Retraining 20 CH-3-PPO agents at the shipped 70k-step configuration and selecting the best by mean return over the 100-point $x_0$ grid gives a best-agent mean of 84.58 against the paper's 98.10 — regret ~14.8 vs the claimed 1.29. Of the 20 seeds, 13 never reach the goal at all; the winning agent hits max 98.74 on some starts but fails outright on others. The sibling CH-3-ARS pipeline reproduces cleanly (best-agent mean 98.95 vs paper 98.74) and CH-6-ARS on Pendulum lands within 0.2% of the paper's row, so the training/evaluation harness itself is sound — this is PPO seed fragility that 20 seeds does not tame at the shipped hyperparameters. Because the paper's 98.10 anchors the CH-3-PPO row of Tables 1/2/4 and the 4.24× PPO regret factor, either the seed budget needs increasing or the selection rule needs to be documented and deterministic.
• Best-agent selection rule is not recoverable from the code. The Mountain Car notebooks hardcode the best-agent filename by an author-side timestamp; reproducing from a clean checkout requires reverse-engineering the selection criterion. The most defensible substitute (argmax of mean evaluation return over the 100 $x_0$ grid) matches CH-3-ARS well but does not resolve CH-3-PPO. Please replace the hardcoded name with a deterministic selection cell.
• Notebook 01 depends on shelve-cache keys no notebook in the repo ever writes. The analytic/comparison notebook that produces $\pi_{\text{ana}}$'s 99.39, every SB3-Zoo baseline mean, and the $L^2$ policy-distance column of Table 4 reads keys analytic_cs_over_x0 and c1_max_1000 that are never written by the shipped code. As a result, $r_{\text{ARS}} = 2.72$, $R(\pi_{\text{ana}}) = 99.39$, and the L2 column could not be recomputed end-to-end from a clean checkout in this session; the notebook has to be repaired by hand.
• Best-of-20 CH checkpoints are not shipped. CH-3-{ARS, REI, PPO} coefficients that back Tables 1, 2, 4 and Figures 5, 10, 11 live only in a gitignored picklesharedb/ directory. This makes the reported factors non-verifiable point-for-point and — as CH-3-PPO shows — this matters when a single retraining run misses the reported mean by ~14 return.
• External polyagents package is an unversioned hard dependency. The multi-variate Chebyshev implementation — the paper's central technical contribution — is not vendored; it is installed from GitHub via a #egg= fragment naming an import package rather than a distribution. Pinning to a commit hash or vendoring the modules would insulate the paper's numbers from upstream drift.
• Several appendix experiments have no shipped driver. The PPO-MLP architecture sweep (Table 3), the CH-3-SAC branch of Table 6, the CH degree sweep of Table 5, the Pendulum best-degree sweep, and the Appendix A.5 discrete-time fmincon verification each lack a runnable script in the released repo. A cell that reads ppo_net_size_episode_results (Table 3) would raise KeyError on a clean checkout; utils/exp_run.py exposes no sac branch for Table 6.
• aero_envs packaging silently drops the compiled plant model. The companion aero_envs package's setup.py declares no package_data and has no MANIFEST.in, so pip install drops resources/linux/model.so. gym.make(...) then fails with a resource-not-found error that looks like missing proprietary data — an earlier reproduction pass wrongly concluded the whole Aero section was unreproducible before tracing it to packaging.
• Aero 2 real-world evaluation is not offline-reproducible and its hardware requirement is undocumented. Table 9 needs the physical Quanser Aero 2 plus the Quanser HIL SDK; aero_envs/real/real_env.py is a 0-byte file and the only real-hardware path talks to a HIL card over TCP. The notebook contains no file-read of any measurement log, so a reproducer cannot tell from the code that Table 9 needs hardware. Publishing per-episode telemetry (pitch, target, action, current) alongside a README note would let referees verify Tables 8/9 and Figure 14 offline.
• Aero 2 Chebyshev/ARS checkpoints are not shipped. Only one Aero PPO MLP checkpoint (aero_best_ppo.zip) is present; the CH-3-ARS, CH-3-PPO, and ARS-MLP Aero policies populating Tables 8 and 9 must be retrained (12 agents × 150k PPO steps + 4M ARS steps) plus two external packages.
• Pendulum baseline loader currently points at the wrong directory. utils/parallel.py hardcodes folder='rl-trained-agents' for pendulum baseline loading, while the intended spec is folder='rlzoo-pendulum-trained-agents'. Both directories exist; the unpatched code silently loads a different agent set. If the authors' own runs went through the unpatched path, the published Pendulum SOTA baselines may correspond to unintended agents — this needs confirmation.
• Analytic-policy constants in code drift from the paper. run_analytic_policy uses $c_1 = 4.3345$ and $c_2 = 4.836$, rather than the paper's 4.3346 and 4.8358. The fourth-decimal drift probably moves the mean return by only ~0.01, but $\pi_{\text{ana}}$ anchors every regret.

Recommended Changes

Essential

• Explain and, if needed, correct the CH-3-PPO reproduction gap. Diagnose why retraining 20 CH-3-PPO seeds at the shipped configuration yields best-agent mean 84.58 rather than 98.10 (13/20 seeds fail to learn). Either (a) increase the seed budget, tighten hyperparameters, or clip the training run so that a fresh reproduction reliably lands near 98.10, or (b) document a deterministic best-agent selection rule that gives 98.10 from the existing 20-seed pool — and replace the timestamp-based hardcoded filename in the notebook accordingly.
• Fix or explain the CH-3-ARS min-vs-mean impossibility. In Tables 1 and 4, $R_{\min} = 98.95$ exceeds mean $R = 98.74$. Re-check the underlying evaluation (the two values may be swapped — a fresh CH-3-ARS run gives min 98.74, mean 98.95), correct the affected cells, and re-verify the 0.65 regret and 4.18× factor.
• State the regret reference point once, explicitly. Add one sentence to the abstract and Section 5 stating that regret is measured against $\pi_{\text{ana}}$'s mean return 99.39 (not the trivial 100 nor $\pi_{\text{opt},x_0}$), so readers can reconstruct the factors and future work can compare consistently.
• Report per-seed variance for the headline Chebyshev rows. Extend Tables 1 and 4 with mean ± std over the 20 seeds actually trained (not only best-of-20), mirroring Table 5. This directly addresses the best-of-20-vs-deployed asymmetry and would have made the CH-3-PPO reproducibility issue visible up front.
• Add a linear-policy ablation. Train a plain linear (or $d=1$ Chebyshev) policy with ARS, PPO, and REINFORCE. Given the analytical optimum is linear in $v$, this ablation is essential to separate "Chebyshev basis wins" from "higher-degree polynomials optimise more easily".
• Ship the best-of-20 Chebyshev checkpoints. Move the CH-3-{ARS, REI, PPO} weights (and the CH-6-ARS / CH-5-PPO Pendulum and CH-3-{ARS, PPO} Aero weights) from the gitignored picklesharedb/ into the released repository so Tables 1, 2, 4, 7, 8 are point-for-point verifiable without re-running the seed lottery.
• Ship the missing sweep drivers and repair notebook 01. Add: (i) the PPO-MLP architecture-sweep script for Table 3, (ii) the CH-degree sweep script for Table 5 and the Pendulum best-degree sweep, (iii) the CH-3-SAC training branch for Table 6 (including the $\sigma$-head convention), (iv) a Python re-implementation (or the Matlab scripts) for Appendix A.5, and (v) cells that produce the shelve keys analytic_cs_over_x0 / c1_max_1000 that notebook 01 currently reads without them ever being written.
• Derive, do not just assert, the worst-case starting position. Add a short argument (or an explicit reference to a numerical sweep) showing that $\min_{x_0}\max_{\pi}$ is attained near $\hat x = -\pi/6$, since $\pi_{\text{ana}}$'s definition hangs on this.
• Clarify the deployed $\pi_{\text{ana}}$. Rephrase the "independent of $x_t$" sentence at Theorem 2.4 to make clear that $x$-independence holds only for the unconstrained optimum, and tie the 4.3346 constant in Section 3 explicitly to the worst-case $x_0$.
• Explain the Aero 2 sim-to-real power reversal. Add a paragraph to Section D.2 addressing why CH-3-* power consumption drops ~3–11× on the real system while PPO's increases, so the sim-to-real narrative for pitch deviation is not undercut by an unexplained reversal in the adjacent column.
• Fix the aero_envs packaging defect and document the Aero 2 hardware requirement. Declare package_data / MANIFEST.in so pip install aero_envs includes resources/linux/model.so, and state in the README that Table 9 (and the sim-to-real portion of Table 2) requires the physical Quanser Aero 2 plus the Quanser HIL SDK. Publish the ten real-episode telemetry logs so referees can verify offline.
• Confirm and pin the Pendulum baseline directory. Reconcile utils/parallel.py's hardcoded folder='rl-trained-agents' with the intended rlzoo-pendulum-trained-agents and confirm which agent set produced the paper's Pendulum SOTA baselines.

Suggested

• Reconcile the parameter-count arithmetic. Fix the footnote so $192 + 4160 + 65 = 4417$ (not $2\cdot 65 + 64\cdot 65 + 65 = 4355$), and state which Chebyshev components enter the 16-coefficient denominator that gives 277.
• State the Pendulum $n$ used for $(d+1)^n$ counts. One sentence clarifying whether Chebyshev on Pendulum uses $n=2$ (angle + velocity) or $n=3$ (cos, sin, velocity) resolves the ambiguity that immediately affects the Limitations discussion.
• Fix the small proof-sheet defects. Replace "cosine being zero" with "cosine being one" in the Lemma 2.3 discussion; re-render the garbled first sentence of that proof; clean up the "reaches the goal fulfills (5)" run-on in the Theorem 2.4 proof.
• Rename Table 5. Change the title from "CH-3-PPO … for different polynomial degrees" to something consistent with the naming convention (e.g., "CH-$d$-PPO").
• Match Table 6's PPO degree to Table 4's, or explain the bump to $d=4$. Otherwise "align with what we have seen with native implementations" reads as a like-for-like comparison it is not.
• Add a caption sentence to Table 2 stating the per-column evaluation protocol (100 $x_0$ for MC, 50×50 grid for Pendulum, 10 target signals for Aero 2 sim and real).
• Note the truncated $x$ range $[-1.2, 0.45]$ for $\lVert\pi - \pi_{\text{ana}}\rVert_2$ (goal at 0.45) explicitly, rather than leaving readers to guess.
• Clarify the $\sigma$-initialisation with one line, e.g. "$\theta^{(\sigma)}_{0,\ldots,0} = 1$, all other $\sigma$-coefficients zero".
• State that Table 3's $[64,64]$ PPO number (94.22) uses 20 evenly spaced $x_0$ so the small discrepancy with Table 1's 93.91 (100 $x_0$) is not read as an inconsistency; soften "match the one reported on RL Baselines3 Zoo" accordingly.
• Tighten the "None solved / RAdam[8] reaches ~60" phrasing by naming the RL-Zoo solved threshold of 90 inline.
• Align the shipped analytic-policy constants (4.836, 4.3345) with the paper (4.8358, 4.3346), or note that these are rounded from a code sweep.
• Recompute the PPO regret-reduction ratio. $5.48/1.29 = 4.248$ rounds to 4.25, not 4.24; either round consistently or note that ratios use unrounded regrets.
• Move the "phase 2 independent of $k$" caveat out of the parenthetical — the statement holds only in the $v_{\text{wall}} = 0$ regime, and readers may over-generalise.
• Vendor or pin polyagents so the paper's numbers do not silently drift with upstream changes.
• Consider softening "the state space is necessarily explored well" to reflect that a goal-reaching trajectory is a 1-D curve rather than a broad exploration of state space.