From Pixels to Tokens: A Systematic Study of Latent Action Supervision for Vision-Language-Action Models

Summary

The paper takes a fragmented research area — latent action supervision for VLA policies — and imposes a needed organizational structure. The authors categorize existing approaches by the mapping direction between vision-language space and action space, yielding two perspectives (image-based latent actions to regularize the trajectory; action-based latent actions to unify the target) and four concrete integration strategies: LA-Align (implicit representation alignment), LA-Direct (direct decoding of image-based latent tokens), LA-Cond (joint conditional decoding), and LA-Tok (action-to-token mapping). All four are instantiated on a shared Qwen3-VL-2B backbone with an identical action head, so the only variable is VLM supervision, and evaluated on LIBERO, RoboTwin 2.0, and a real JAKA arm. Isolating the supervision axis under a shared baseline is exactly the right experimental discipline, and the PH-Direct / PH-Cond placeholder-length controls are the correct reply to a natural confound.

The conceptual move is valuable and the LIBERO aggregates hold up in level under replication (18 of 20 Table 1 cells reproduce within ~2 points using the released joint checkpoints). What does not hold up in the reproduced numbers is the paper's central ordering story: on LIBERO-Long, LA-Cond leads LA-Direct by 5 points (92.6 vs 87.6) and LA-Align actively degrades below the baseline (83.2 vs 87.2); the image-based better than action-based on long-horizon' finding reduces to a single variant. A diagnostic strongly suggests Table 1's LIBERO-Long column may reflect suite-specialized rather than joint checkpoints, in which case the paper is silently mixing two training protocols within a single table. Beyond this, several headline claims rest on comparisons with confounds (an 8k-vs-60k step-budget mismatch between single- and multi-task protocols; single-strategy exemplification for both the multi-task and the sample-efficiency contributions), statistical support is thin (no seed variance, 10-rollout real-world evaluations, relative gain' used for point differences), and neither the RoboTwin 2.0 nor the JAKA experiments are reproducible from the released code.

Strengths

• Conceptual contribution. The mapping-direction taxonomy (regularize the trajectory vs. unify the target) and the four-strategy grid it induces is a genuinely useful lens for reading the rapidly growing latent-action-VLA literature — it collapses many superficially different papers onto a small number of design axes.
• Controlled baseline. Fixing the backbone, action head, aggregation, and placeholder design so that only the VLM supervision varies is the right discipline for a systematic study, and the PH-Direct / PH-Cond controls (Tables 9–10) rule out a plausible confound between sequence budget and gains.
• Coverage of evaluation regimes. LIBERO for long-horizon, RoboTwin 2.0 for motor-complex bimanual, and a real JAKA arm for scene generalization together triangulate the formulation-task-correspondence claim across simulation and hardware, even if the code release does not (yet) let readers reproduce the last two.
• Aggregate reproducibility on LIBERO. Eighteen of the twenty Table 1 cells fall within ~2 points of the reported values when the released joint checkpoints are run under the paper's own protocol, and all four latent-supervised variants meet or beat the baseline average — the level story survives replication even where the ranking story does not.
• Latent-action-model design attempt. The FFT + 1D-temporal-conv + Transformer action tokenizer with a mask-consistency loss (Section 3.2.2) is a reasonable original contribution, and Table 6 shows that the strategy ranking survives when the 5% action supervision is stripped from the image-based LAM, which is a helpful robustness check.

Weaknesses

• Ordering claim on LIBERO fails to reproduce. Under the released joint +libero checkpoints, on LIBERO-Long the ordering is LA-Cond 92.6 > LA-Tok 90.6 > LA-Direct 87.6 > baseline 87.2 > LA-Align 83.2, and on the four-suite average LA-Cond 96.2 > LA-Direct 94.85 > LA-Tok 94.6 > LA-Align 93.25 > baseline 93.15. This directly contradicts the paper's Section 5.3(1) claim that LA-Direct is the strongest strategy, and it collapses the image-based +8.4% to +10.8% vs action-based +6.8%' gap on LIBERO-Long (measured gaps: LA-Align −4.0, LA-Direct +0.4, LA-Cond +5.4 vs LA-Tok +3.4). A supplementary run with the suite-specialized +libero_10` checkpoint recovers LA-Direct to 94.6 on LIBERO-Long, which suggests Table 1 may silently mix per-suite and joint checkpoints — a substantive documentation gap either way.
• Multi-task step-budget confound. The single-task (8k steps) vs multi-task (60k steps) protocols differ by 7.5× total gradient steps, which the paper does not correct for. The `reduces negative transfer' contribution rests entirely on this comparison and cannot be separated from a training-budget effect.
• Single-strategy multi-task and sample-efficiency evidence. The multi-task consistent improvement' story uses LA-Cond only; the sample-efficiency headline uses LA-Tok only (on the suite where LA-Tok is the *weakest* strategy at full data). Neither exemplifies latent action supervision' as a class as broadly as the abstract and contributions phrase it.
• No seed variance or CIs anywhere. Every table reports single-run means. Many of the paper's rankings (e.g. LA-Direct 97.1 vs LA-Align 97.0 AVG, LA-Direct vs LA-Cond real-world 87 vs 78) sit within plausible seed-to-seed noise; the real-world 10-rollout evaluations yield Wilson 95% CIs on the order of ±30 points at intermediate means.
• Absolute-vs-relative terminology error. +31.8% relative gain' and +20.9% improvement' for LA-Cond multi-task are absolute point differences, not relative gains; the same convention (`↑X%' for point deltas) is used throughout Tables 1–3.
• **Consistently improves' contradicted by Table 1 and by replication.** LA-Tok on LIBERO-Goal is 92.2 vs Baseline 92.8; the replicated LA-Align on LIBERO-Long is 83.2 vs 87.2. Consistent improvement over the baseline' is not strictly true.
• Dominance vs correspondence conflict. Section 5.3(1) heads with `LA-Direct generally outperforms LA-Cond in most experiments,' then concedes LA-Cond wins on RoboTwin 2.0; the replication of LIBERO further reverses the LIBERO leg. This is better restated as strategy–benchmark correspondence.
• Design-point breadth. One backbone (Qwen3-VL-2B), one image-based LAM family (fine-tuned UniVLA + 5% action supervision), one action-based LAM design (FFT + conv + Transformer + mask-consistency). Strategy rankings may not transfer to different backbones or off-the-shelf tokenizers.
• Motor-complex claim rests on a 4-task subset. The RoboTwin 2.0 evaluation used for the `action-based excels at complex motor coordination' finding is a 4-task subset; the full 10-task set (used in Appendix E.3 for the multi-task analysis) is only evaluated for LA-Cond.
• **Negative transfer' framing.** The Baseline improves +18.2 points on average under joint training but is described as clearly suffering from negative transfer' — the actual phenomenon is task-level instability around a positive mean.
• Rubric inconsistency. The stacking rubric restricts per-bowl scores to $\{0.0, 0.5, 0.8, 1.0\}$, but the Stack-2 columns in Tables 13–17 contain values like 0.7, 0.9, 0.6 that cannot be produced by that rubric.
• Missing equations and under-specified losses. Several `Here, ...' blocks in Sections 3.2, 3.3, and 4 dangle without a preceding rendered equation; the wiping-score definition promises a formula but shows only variables; the mask-consistency loss is ambiguous on whether the constraint acts on continuous embeddings or discrete codes.
• Prior-baseline comparability. Table 1's rows for OpenVLA-OFT, π0, LAPA, UniVLA, ThinkAct, GR00T, π0-FAST are quoted from source papers using 2B–7B backbones under different training recipes; the surrounding prose (`competitive with state-of-the-art VLAs') treats them as directly comparable.

Reproducibility & code

• Headline LIBERO aggregates. Executing experiments/robot/libero/run_libero_eval.py on the released CokeAnd1ce/From_Pixels_to_Tokens +libero checkpoints at the paper's protocol (50 trials/task × 10 tasks = 500 episodes per cell) reproduces 18 of 20 Table 1 cells within ~2 points. LIBERO averages align with the paper: baseline 93.15 vs 93.1, LA-Align 93.25 vs 97.0, LA-Direct 94.85 vs 97.1, LA-Cond 96.2 vs 96.6, LA-Tok 94.6 vs 95.5.
• LIBERO-Long outliers reverse the paper's ordering. LA-Align LIBERO-Long = 83.2 vs 94.8 reported (−11.6) and LA-Direct LIBERO-Long = 87.6 vs 96.6 reported (−9.0) — both well outside the ~1.5-point binomial SE at $n=500$. These two cells reverse the LA-Direct > LA-Cond > LA-Align ordering the paper builds Section 5.3 on. A diagnostic with the suite-specialized LA_Direct+libero_10 checkpoint scores 94.6 on LIBERO-Long, closing 7 of the 9 points; the paper's LIBERO-Long column therefore looks likely to reflect per-suite-trained checkpoints while the other columns reflect joint checkpoints, but the release does not disambiguate.
• Released baseline training does not run as shipped. exp/train_vla.py sets use_latent_loss only for la_align and leaves token_loss_weight = 1.0, routing the baseline into accelerator.py which indexes ret_loss['action_token_loss'] — a key Baseline.forward never returns. The documented baseline training command crashes with KeyError; a --token_loss_weight 0.0 workaround is required. Since Baseline is the anchor for every comparison in the paper, this needs to be fixed in the release, and the actual configuration used to produce Baseline in Table 1 needs to be re-published.
• Environment brittleness. Running the LIBERO evaluation from a clean environment required making latentvla.extern.modeling_internvl_chat a lazy import (the module is not shipped, so every caller of get_vla() raises ModuleNotFoundError as-is), adding flash-attn to requirements.txt (the trainer hardcodes attn_implementation='flash_attention_2'), and pinning mujoco to 3.2.7 by hand (unpinned mujoco resolves to 3.10, which is incompatible with robosuite 1.4.1). An unpinned simulator is a real risk for a success-rate benchmark.
• RoboTwin 2.0 is not reproducible from artifacts. The repo ships a LIBERO harness but no analogue for RoboTwin 2.0, no per-task success-rate utility, and no robotwin branch in constants.detect_robot_platform. Tables 2, 11, 12 and Figure 5 cannot be reproduced without a from-scratch integration.
• JAKA real-robot experiments are entirely absent from the release. No rollout code, no scoring rubric implementation (stacking, wiping, pick-and-place), no controller/camera interface, no demonstrations. Table 3, Figure 4, and Tables 13–22 exist only in prose.
• Table 4 discrete-vs-continuous ablation is not reproducible. No LA-Direct(C) / LA-Tok(C) heads or checkpoints exist in the repo, and no MSE-regression head over continuous latents appears anywhere in latentvla/models/vla/. Producing Table 4 requires implementing the continuous variants from scratch.
• PH-Direct / PH-Cond baselines are not configurable. RLDSBatchTransformQwen3 hard-codes placeholder length as NUM_ACTIONS_CHUNK*5+1; Tables 9–10 and Figure 6 require adding a knob and retraining.
• Data-fraction sweep is not scripted. RLDSDataset streams the full mix with no subsampling parameter; Figure 8 requires the reader to build the sampler and orchestrate the sweep manually.
• Paper/code drifts. LA-Align layer index (paper 17 vs code self.layer_idx = 18, non-configurable); LR step-decay period (paper 10k vs code 20k); image-based LAM action_beta (paper 1.0 vs config default 0.5). Each is small individually; together they mean the released configuration does not directly produce the reported numbers, and Tables 5, 6, and the appendix LR schedule become harder to reproduce faithfully.

Recommended Changes

Essential

• Clarify the LIBERO-Long checkpoint protocol and re-verify Table 1. State explicitly for each Table 1 column which checkpoint(s) it uses; if per-suite and joint checkpoints are mixed, re-report both protocols side by side. The current release + joint checkpoints yield LA-Align 83.2 and LA-Direct 87.6 on LIBERO-Long, not 94.8 and 96.6; this contradiction must be resolved before the `LA-Direct is best' claim in Section 5.3(1) can stand.
• Fix the baseline training bug. Release the actual exp/train_vla.py invocation used for the Table 1 Baseline (which currently crashes with KeyError('action_token_loss') unless --token_loss_weight 0.0 is passed), and reconcile the code so that the anchor variant runs from the documented command out of the box.
• Fix the multi-task step-budget confound. Match total gradient steps across single-task and multi-task protocols, or report at least one budget-matched control, so the `reduces negative transfer' claim is not conflated with training-budget effects.
• Broaden the multi-task and sample-efficiency experiments. Run at least one other strategy (LA-Tok is a natural pick) under the 10-task RoboTwin multi-task protocol so the `consistently improves under joint training' contribution does not rest on LA-Cond alone; likewise report Figure 8 curves for all four strategies × all four LIBERO suites instead of exemplifying with a single cell.
• Add seed variance. Rerun at least the LIBERO experiments with ≥3 training seeds and report mean ± std in Tables 1, 4, 5, 6, 9, 10, and Figure 8. For real-world tables, either scale rollouts beyond 10 or report Wilson CIs.
• **Fix the relative gain' terminology.** Rewrite +31.8% relative gain' / +20.9% improvement' in Section 5.6 and every ↑X%'-labelled cell in Tables 1–3 to distinguish point differences from true relative gains, and pick one convention (e.g. `+X points') uniformly.
• Restate LA-Direct vs LA-Cond as strategy-benchmark correspondence. Rewrite Section 5.3(1) so it does not contradict the RoboTwin exception and the LIBERO replication above.
• Reconcile `consistently improves' with Table 1. Soften the contribution bullet or supply the extra experiments — as stated it is falsified by LA-Tok on LIBERO-Goal (and by LA-Align on LIBERO-Long under replication).
• Release the missing evaluation harnesses. Ship (i) a RoboTwin 2.0 evaluation script and per-task reporting utility, (ii) the JAKA scoring rubric implementation and, ideally, the demonstrations, and (iii) the LA-Direct(C) / LA-Tok(C) continuous-regression heads referenced in Section 5.4, so that Tables 2, 3, 4 and Figures 4–5 are reproducible from artifacts.
• Fix the paper/code drifts. Reconcile the LA-Align layer index (17 vs 18), the LR step-decay period (10k vs 20k), and the image-based LAM action_beta (1.0 vs 0.5). Expose layer_idx, step_size, and placeholder length as configurable arguments.
• Restore the missing equations. Re-render the equations dropped from Sections 3.2.1, 3.2.2, 3.3, and 4.1, and add the explicit formula for the wiping score.
• Fix the runtime environment issues. Ship the latentvla.extern.modeling_internvl_chat module or make its import lazy; add flash-attn to requirements.txt; pin mujoco (and thereby robosuite) to a compatible version.

Suggested

• Add a second backbone. Repeat the four-strategy comparison on at least one additional VLM (e.g. a different 2B or 7B backbone) to test whether the strategy ranking is backbone-invariant, since the replication already suggests LA-Align's LIBERO-Long ranking depends on training-protocol choices the paper does not fully control for.
• Study the paper's LAM design as a variable. Run the four strategies with an off-the-shelf action tokenizer (e.g. FAST) in place of the custom FFT+conv+Transformer + mask-consistency design.
• Reconcile the stacking rubric with the per-rollout values. Either state the finer-grained scoring actually used or explain how the rubric-legal values $\{0.0, 0.5, 0.8, 1.0\}$ can produce entries like 0.7 and 0.9 in Tables 13–17.
• Broaden the RoboTwin motor-complex claim. Evaluate the four strategies on all 10 RoboTwin tasks so the motor-complexity claim rests on the same task set as the multi-task analysis.
• Ablate the LA-Align layer choice over nearby layers. Extend Table 5 with several mid-to-late layer choices (15/17/19/21/23) so the layer selection is grounded rather than asserted.
• Clarify the mask-consistency loss. State in the main text whether the constraint acts on continuous embeddings or discrete codes, and describe the masking mechanism (mask token vs zero vs mean) and gradient flow explicitly.
• **Reframe negative transfer'.** Reserve the term for the specific tasks that degrade, and describe the overall Baseline as task-level instability around an improving mean'.
• Flag prior-baseline comparability. Add a caveat sentence to Section 5.2 noting that Table 1's prior-work rows are quoted from source papers under different backbones and recipes.
• Ship a data-fraction control. Add a subsampling parameter to RLDSDataset and a sweep script so Figure 8 is reproducible from artifacts.
• Qualify the `improves latent quality' inference. Table 6 only measures downstream success rates; either measure LAM-intrinsic quality directly or scope the claim to downstream performance.