XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representations

Summary

The paper introduces XR-1, a Vision-Language-Action (VLA) framework whose central contribution is a shared discrete latent — Unified Vision-Motion Codes (UVMC) — learned by a dual-branch VQ-VAE that jointly encodes future-frame visual dynamics and robot action / proprioception. UVMC is then used as a supervisory target for a VLM+action-head policy in a three-stage pipeline (self-supervised UVMC learning; large-scale UVMC-guided policy pretraining; task-specific fine-tuning). The conceptual move — sharing a codebook across a vision branch and a motion branch, tying them with a cross-modal alignment loss, and having a VLM predict tokens in that shared space — sits neatly between two prior lines (action-tokenization à la BeT/QueST/Moto and vision-latent lines à la LAPA/VPP/GO-1) and is a plausible way to squeeze complementary signal out of heterogeneous data sources, including actionless human video. The experimental campaign is ambitious: six real robot embodiments, 120+ tasks, ~14k rollouts, plus a CALVIN D$\rightarrow$D result and multiple ablations. On the reported numbers, XR-1 clearly leads $\pi_0$, $\pi_{0.5}$, RDT, UniVLA, and GR00T-N1.5 by wide margins on every embodiment table, and it degrades gracefully to a lightweight XR-1-Light variant. The main conceptual limitation is that the paper's mechanistic story (the shared codebook plus the alignment loss are what buys the gains) is only loosely isolated by the ablations — the alignment loss is co-varied with the presence of both branches, the Stage-1 data-scaling ablation confuses Stage-1 and Stage-2 data volume, and the alignment loss itself is not written out in the main text (its posterior $q(\cdot)$ is left undefined). The methodological limitation is that every headline comparison — including narrow ones like the 1.7-point "on-par" claim and the 5.8-point Ego4D ablation — is reported as a single point estimate over 20 rollouts, with no per-seed variance, no confidence intervals, no blinding of the human success evaluator, and no clearly documented baseline-training budget. Executing the released code confirmed that Stage-1/2/3 XR-1 training runs (after some patches) but that the baselines, the CALVIN wrapper, the XR-1-Light backbone, the per-embodiment rollout harness, and the interpretability-analysis scripts are absent; the one paper number our replication could actually measure (inference throughput) came out ~$10\times$ below the paper's, and the qualitative retrieval / t-SNE claims did not reproduce on the released sample.

Strengths

• Conceptual contribution. Encoding vision dynamics and robotic motion in a shared discrete codebook, with an alignment loss regularizing visual codes toward motion codes, is a clean way to unify the two dominant streams of latent-representation work (visual-dynamics and action-tokenization). The design allows action-free human video to contribute via the vision branch while robot data trains both branches — a genuinely useful cross-source recipe.
• Scale and breadth of empirical study. Six heterogeneous embodiments (Tien Kung 1.0/2.0, Single/Dual-Arm UR-5e, Dual-Arm Franka, AgileX Cobot Magic V2.0), 120+ real tasks, ~14k rollouts, plus CALVIN D$\rightarrow$D and multiple ablations. Very few VLA papers report at this scale across this many real platforms; the tables convey a consistent story across embodiments seen and unseen during pretraining.
• Model-agnostic instantiation. Demonstrating UVMC on both a large PaliGemma-based XR-1 and a lightweight Florence-2/SwitchVLA XR-1-Light shows the codebook does not require a specific action-head architecture, and the Table 3 delta (+15 pp on the lightweight variant) is a compelling data point.
• Cross-embodiment transfer signal. The Table 16 experiment showing that training identical skills across three embodiments raises per-embodiment performance is a nicely targeted probe of the "embodiment-agnostic" claim, and the qualitative retrieval / t-SNE analyses (Figs. 12–15) are consistent with the claim that UVMC captures action semantics that transfer across morphology (though this signal is not reproducible from the shipped sample; see Reproducibility & code).
• Deploy path exists. The deploy/real_robot/xr1_deploy.py + xr1_inference.py code is present and can be exercised end-to-end on the Dual-Arm Franka and Dual-Arm Tien Kung 2.0 paths, providing a concrete artifact for downstream use.

Weaknesses

• Alignment loss under-specified. The vision-motion alignment loss is a stated core novelty and the mechanism by which "human-only demonstrations [are] mapped into the robot's action space", yet the equation is not fully rendered in the main text and the posterior $q(\cdot)$ is described only as "the posterior distribution in the codebook space" — the choice between soft categorical, one-hot argmin, or an embedding-space $\ell_2$ fundamentally changes the training dynamics and reproducibility.
• UVMC framed as 'discrete' but supervised with continuous MSE. The Stage-2 target is the concatenated quantized embedding and the loss is a plain squared $\ell_2$ regression on that vector. Nothing forces $F(l,o,t)$ to lie on the codebook, and the paper does not describe a nearest-neighbor snap at inference or an ablation against a codebook-index cross-entropy head, which is the more standard choice in the discrete-latent-VLA literature.
• Weak isolation of the alignment loss in ablations. Table 3 Exps 3–5 co-vary the alignment loss with the presence of both branches, and Table 14 Exps 7–8 ablate motion-only / vision-only codes rather than the alignment term itself. As a result the specific contribution of the vision-motion alignment (vs. simply having two codebooks) is not cleanly quantified.
• Asymmetric reconstruction targets not motivated or ablated. The vision branch is trained to predict a single future frame $c_{t+h}$ while the motion branch reconstructs the full action sequence $a_{t:t+h}$. This is a strong asymmetry with plausible consequences for how the shared codebook allocates capacity, and it is neither justified nor ablated (nor is the horizon $h$).
• Vision encoder fully fine-tuned without a forgetting check. SigLIP is left fully trainable in Stage-1 — a substantial share of the effective "vision" representation is being re-optimized on ~110M frames. Whether this degrades SigLIP's language-alignment (relied on again by PaliGemma at Stage-2), and whether the gains actually come from the codebook or from adapting SigLIP to robotics footage, is neither ablated nor discussed.
• Reporting without variance or blinding. All success rates are single point estimates over 20 rollouts per task. With standard binomial CIs, several headline gaps (the "on par" 65.0 vs 66.7 at Exp 9/5, the +5.8 pp Ego4D ablation, the +15% cross-embodiment gain) sit within a single 95% CI. Success is judged by human evaluators without any statement about blinding to the method, which is a load-bearing assumption for the qualitative-verdict tables.
• Baseline provenance and budget not documented. The five baselines drive the paper's headline story yet the training / fine-tuning budget, checkpoint origin, and any hyperparameter tuning are only partially disclosed (the $\pi_0$ JAX vs LeRobot note is helpful; nothing analogous is said for $\pi_{0.5}$, RDT, UniVLA, GR00T-N1.5). Given 30–40 point gaps this materially limits how the gaps should be interpreted.
• Scaling story mislabeled. The 1%-to-100% scaling ablation is labeled as "Stage-1 pretraining data ... without any subsequent fine-tuning", but Stage-1 in this paper's own definition trains no action head — the reported success rates require Stage-2. Either the label is wrong, or the setup differs from the Section 3 definitions; either way the reader is left unable to interpret what is actually being scaled.
• Several claim–evidence mismatches. "XR-1 consistently outperforms $\pi_0$ across all settings" for Table 4 is at best ambiguous under the rendered layout; "baselines collapse to near 0%" is factually incorrect for SUR-StackPlates and AGX-CloseToolbox where several baselines are 55–90%; "the evaluation setting is more favorable to the baselines" in fast-adaptation ignores the very large pretraining asymmetry that favors XR-1.
• Task count and dataset totals are internally inconsistent. "over 120 tasks", "123 tasks", "the 120 real-world tasks", and "20 tasks per embodiment $\times$ 6 embodiments = 120" coexist without reconciliation. Table 8's top-level frame counts sum to 164.1M despite the paper elsewhere reporting ~110M, and the sub-dataset "Weight" column proportions do not track either the frame or episode columns.
• Notation and unit slips. Motion decoder equation uses $z_{mo}$ where the prose says the quantized $z^e_{mo}$ is used; motion-branch parameter count is 34M in text but 33M in Table 5; vision encoder / decoder ratio is claimed "roughly five times" but the numbers give ~$4.6\times$; motion-decoder loss is called "diffusion-based" in text but "flow matching" in both tables; "Unified Vision-Motion Condition" appears once instead of "Codes"; chunk-rate $\times$ chunk-length arithmetic gives 250 Hz, not the stated 200 Hz.
• Overclaimed generality of the framework. The phrase "integrates with diverse VLA architectures" is supported by two backbones ($\pi_0$-style PaliGemma and Florence-2/SwitchVLA), both of the VLM + action-head shape — a narrower demonstration than "diverse" implies.
• Cross-embodiment fairness. The Table 16 argument equates 6 same-embodiment tasks with 2 skills $\times$ 3 embodiments = 6 cross-embodiment tasks and calls the data volume "equivalent", but with the trajectory-count variance visible in Table 19 that equivalence is not obvious and is not backed by frame counts.

Reproducibility & code

We executed the shipped repository (https://github.com/Open-X-Humanoid/XR-1) end-to-end on an A100 80 GB host. Stage-1 training runs after four moderate patches (see below). Stage-2 and Stage-3 training paths import and instantiate. The XR-1 policy factory exposes xr1_stage1, xr1_stage2, and pi0; ACT and Diffusion Policy raise NotImplementedError. Beyond this, most artifacts the paper's tables rest on are absent, and one number we could actually measure did not match.

• Headline aggregates cannot be re-run. Tables 2, 9–13 and Figures 4/6/7 require both physical robots + human success scoring and the five VLA baselines. The repo implements none of $\pi_{0.5}$, RDT, UniVLA, or GR00T-N1.5 (0 grep hits), and there is no rollout / success-rate harness for any of the six embodiments — the only success_rate references live in upstream LeRobot's gym evaluator, which cannot construct an XR-1 policy on a real embodiment. Deploy handlers exist only for Dual-Arm Franka and Dual-Arm Tien Kung 2.0.
• Inference throughput did not reproduce. Timing Xr1Stage2Policy.select_action (action_horizon = 50) with warmed CUDA on an A100 80 GB PCIe gave ~0.44 Hz chunk / ~22 actions/s, roughly $10\times$ below the paper's ~5 Hz / ~200 actions/s. Since the A100 is generally faster than an RTX 4090, the gap is not hardware — the shipped path hardcodes fp32 (a bf16 autocast probe reached ~0.60 Hz), and the 10-step flow-matching sampler is kernel-launch-bound. Even the fastest single iteration observed was ~0.64 Hz. Whatever software stack produces the paper's numbers is not documented in the released repo.
• UVMC interpretability claims did not reproduce on the released sample. Re-running the frame-to-frame retrieval with the released Stage-1 encoder over XR-1-Dataset-Sample produced modes SE.ST. = 0.960, DE.ST. = 0.959, DE.DT. = 0.966 — essentially flat, with DE.DT. actually the highest mode instead of the lowest, and SE.DT. unpopulated because the sample carries only one task per embodiment. The paper's 0.97 / 0.82 / 0.60 / 0.46 graceful-degradation pattern is not observable. The t-SNE by task/embodiment produces well-formed clusters, but the Franka/UR CleanTable overlap Figure 15 highlights cannot be shown from the sample (no shared task across embodiments); silhouette and AMI actually run slightly against the task-over-embodiment claim. This is a released-artifact limitation as much as an evidentiary one — the paper does not ship the exact retrieval task list of Appendix H.
• CALVIN state-of-the-art claim is not backed by shipped code. No CALVIN environment integration, adapted checkpoint, per-subtask evaluator, or config that would emit Table 17 (4.256 vs 3.885) is present; a grep for calvin across .py and .sh files returns zero hits. Because CALVIN is a standardized public benchmark, this is a fairly cheap gap to close.
• XR-1-Light backbone is not in the released repo. Despite anchoring the "+15% absolute" Table 3 rows and the lightweight-model narrative, no code for Florence-2 or SwitchVLA integration ships. Exp. 1 and Exp. 2 of Table 3 are not re-runnable as-shipped.
• Ablation grids described but not scripted. The 1%/10%/50%/100% Stage-1 data-scale sweep in Table 3, the codebook geometry sweep in Table 14, the Ego4D ablation in Table 15, and the SelfRobot vs CrossRobot experiment in Table 16 are in-principle configurable via existing knobs (the KL alignment is gated by kl_loss_w, and Ego4D masking exists via no_ego4d_mask in modeling_xr1_stage1.py), but no subsampled data manifests, per-experiment configs, or "DT" (direct-training) mode ship.
• Interpretability figures have no analysis scripts. No code exports UVMC codes to disk or computes retrieval / similarity / t-SNE for Figs. 12–15 or Table 18. Given the codes are computed internally by the Stage-1 model, this is easy to add.
• Task manifests not machine-readable. Table 19 enumerates tasks by name and instruction, but no JSON/YAML manifest, scene reset spec, or success criterion definition is shipped; Appendix K.1 is referenced but not present as a file. This limits cross-verifiability of the evaluation setup.
• Shipped training path has moderate defects. The floating torch/torchvision pin resolves to a torchvision that removed torchvision.io.VideoReader, breaking every training entry point at import; the default dataloader worker count does not fit small-CPU environments; the Stage-3 --debug smoke test inherits steps=100000 and save_freq=20, writing ~167 GB of checkpoints in 48 minutes without printing a single loss line; and the UVMC loss decomposition is forwarded only to wandb and silently dropped when wandb is disabled. These are moderate — none blocked replication in principle — but each requires a manual patch.

Overall reproducibility risk is high on the head-to-head baseline comparisons, the CALVIN result, and the XR-1-Light ablation; moderate on the per-embodiment averages (feasible with real hardware plus the missing manifests); moderate on the paper's inference-throughput number and the interpretability figures (where our execution actively disagrees with the paper); and low on the XR-1 training pipeline itself, once the shipped defects are patched.

Recommended Changes

Essential

• Add per-cell variance and blinding disclosure. Report per-task success as $p \pm \text{binomial-CI}$ over the 20 rollouts, run 2–3 seeds for each headline aggregate (per-embodiment averages, the four Table 3 scaling points, the Ego4D ablation, the cross-embodiment Exp. 1/2), and state whether the human success adjudicator was blinded to the method behind each rollout. This addresses the "Reporting without variance or blinding" weakness and reframes the 1.7-point "on-par" and 5.8-point Ego4D claims as either significant or not.
• Document baseline provenance and training budget. Add a table listing, for each of $\pi_0$, $\pi_{0.5}$, RDT, UniVLA, GR00T-N1.5: checkpoint source (paper release vs. re-implementation), pretraining data seen, downstream fine-tuning schedule, and whether hyperparameters were tuned on the 20-task benchmark. Required to interpret the 30–40 pp gaps in Tables 2, 9–13.
• Write out $L_{\text{align}}$ and specify $q(\cdot)$. In Section 3.2 (or a clearly-referenced appendix), give the exact expression for the alignment loss and define the codebook posterior — soft categorical, hard one-hot, or otherwise. Addresses "Alignment loss under-specified".
• Add an ablation that isolates the alignment loss only. Two branches + no alignment vs. two branches + alignment, everything else fixed, on the same 6-DUR benchmark. Directly resolves "Weak isolation of the alignment loss in ablations".
• Fix the Stage-1 scaling ablation labels or setup. Either rename the scaled axis to what it actually is (the joint pretraining corpus feeding Stages 1 and 2), or re-run the sweep with Stage-2 held fixed and Stage-1 data varied so the current label is honest. Fixes "Scaling story mislabeled".
• Release the missing pieces of the code. In particular: baseline wrapping instructions (versions / commits / configs), a CALVIN wrapper reproducing Table 17, the XR-1-Light Florence-2/SwitchVLA training code, a per-embodiment rollout harness with task manifests as JSON, dumping utilities for UVMC codes plus the retrieval / t-SNE analysis scripts, and the exact retrieval task list of Appendix H. Together these close every gap flagged in Reproducibility & code except the shipped-training-path defects (see below).
• Document — or fix — the inference-speed pipeline. State the exact autocast / torch.compile / driver / thread-pinning configuration needed to reach ~5 Hz chunk / ~200 actions/s on an RTX 4090, or revise the numbers to reflect the shipped fp32 configuration. Addresses "Inference throughput did not reproduce".
• Patch the shipped training path. Pin torch / torchvision to a version that keeps torchvision.io.VideoReader, cap --debug steps/save frequency for smoke tests, and add a stdout/JSONL metrics sink for the UVMC loss decomposition so wandb is not a hard dependency. Addresses the "Shipped training path has moderate defects" point.

Suggested

• Reconcile the 120 vs 123 task count and the ~110M-vs-164M frame totals. Pick a single number for each and use it throughout, and either enumerate the extra three tasks or drop the "123" figure. Addresses "Task count and dataset totals".
• Clarify the Table 8 sub-dataset "Weight" column. State whether the sub-weights are epoch-level sampling weights, effective frame ratios, or something else, and either make them sum coherently with the frame/episode columns or annotate the discrepancy. Fixes the second half of the same weakness.
• Add a UVMC-supervision decomposition ablation. Clarify (and ablate) whether Stage-2's gain comes from the pretrained code alignment, from the [Z_MO] / [Z_VIS] token prediction, or from vanilla Stage-2 policy pretraining without UVMC targets. Complements "Weak isolation of the alignment loss".
• Fix the notation / unit slips. In particular: $z^e_{mo}$ vs. $z_{mo}$ at the motion decoder, motion-encoder 34M vs. 33M, "roughly five times" vs. the actual ~$4.6\times$ ratio, "diffusion-based" vs. "flow matching" text/table conflict, "Unified Vision-Motion Condition" vs. "Codes", and the $5 \times 50 = 250$ vs. 200 chunk-vs-action-rate arithmetic. Directly addresses "Notation and unit slips".
• Soften or narrow the mismatched claims. For Table 4 replace "consistently outperforms $\pi_0$ across all settings" with a per-cell comparison or an averaged statement; for SUR-StackPlates and AGX-CloseToolbox pick tasks where "near 0%" actually holds; for fast-adaptation describe both asymmetries (per-task specialization vs. pretraining) instead of only the one favoring baselines. Fixes "Several claim–evidence mismatches".
• State Stage-3's scope in one place and stick to it. Either it is task-specific post-training for deployment (Section 3.4), or it is held-out generalization testing (Section 4.1); use the same definition throughout so XR-1-oob is unambiguously "Stage-2 without Stage-3" of a single kind.
• Rename or clarify the "AGX-StoreButton" and "control box"/"distribution box" mismatches so each representative example maps to a specific row in Tables 11 / 19. Small fix but improves cross-referenceability.
• Soften the "diverse VLA architectures" wording until UVMC is demonstrated on an architecturally distinct third backbone (e.g. a fully autoregressive discrete-token VLA), or add such a demonstration.
• Report frame-level equivalence for the Table 16 cross-embodiment experiment, or match SelfRobot / CrossRobot conditions on total trajectories or frames rather than tasks. Addresses "Cross-embodiment fairness".
• Motivate or ablate the asymmetric reconstruction targets. Justify one-frame-vs-full-sequence reconstruction and ablate the horizon $h$, or note the design explicitly as a hyperparameter choice.
• Report a frozen-SigLIP ablation. Confirm the codebook is doing the work, not simply the fine-tuned vision backbone. Addresses "Vision encoder fully fine-tuned".