Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

Summary

This paper introduces Video-based Optimal Transport Preference (VOTP), a semi-supervised approach to offline preference-based RL that embeds trajectory segments with a pretrained video foundation model (S3D on HowTo100M) and uses an entropic optimal-transport plan between the segments in a very small labeled preference set and each unlabeled candidate pair. Preferences are then propagated by a bilinear aggregation of the OT couplings weighted by the labeled preference matrix $R$, thresholded by $\tau_P$, and the pseudo-labels are combined with the labeled pairs to train a reward function used by IQL. The conceptual move is elegant: rather than trusting an under-trained reward model to bootstrap pseudo-labels (SURF's failure mode) or synthesizing new trajectories (FTB's expensive route), VOTP treats perceptual alignment in a rich pretrained space as the carrier of preferences. Evaluations span 2 D4RL locomotion tasks, 4 MetaWorld tasks, a visual-distractor robustness study, encoder and cost-metric ablations, a real-robot manipulation setup, and reward-correlation diagnostics that offer a plausible mechanistic story. Central limitations are conceptual and evidential rather than fatal. Theoretical scaffolding around the OT layer is underspecified — the $S_{\max}$ derivation that guarantees $S_{\text{norm}}\in[-1,1]$, the mapping from sign(S_norm) back into the $\tilde y\in\{0,1,0.5\}$ label convention, and the treatment of ties are informal or absent, and the algorithmic choice $M=2$ reduces each Sinkhorn call to a per-pair matching that is much smaller than the trajectory-level framing suggests. Several claims (high-fidelity pseudo-labels, robustness, feedback efficiency) are generalized from narrower evidence than framed — sweep-into's 67% pseudo-label accuracy, only two tasks in the robustness table, and a headline table where many margins sit within 5-seed noise. Finally, the SIM ablation does not clearly isolate OT's contribution from the shared ViFM encoder, and key reproducibility levers (Sinkhorn $\varepsilon$, per-task $\tau_P$, VOTP source, real-robot demonstrations and labels, and the distractor catalog) are not released.

Strengths

• Conceptual contribution. Reframing pseudo-labeling as perceptual alignment in a pretrained video space, then reading preferences off the transport plan, is a genuinely new lever for offline PbRL and side-steps SURF's confirmation-bias failure mode by decoupling pseudo-labels from the reward model being trained.
• Diagnostic story. The reward-correlation figures (Figures 6, 17) and the pseudo-label accuracy table (Table 13) provide a mechanistic explanation for the downstream gains rather than only reporting outcomes, and the correlations for VOTP are systematically higher than P-IQL across most tasks.
• Compute story vs FTB. The comparison against FTB (~2 days per run) at ~2 hours per VOTP run is a practically meaningful contrast, and it is credible given that the OT plan is a small linear-algebra step relative to a diffusion-based trajectory generator.
• Encoder ablation is informative. Comparing IFMs (R3M, CLIP) against ViFMs (S3D, VideoCLIP, InternVideo) at fixed pipeline (Figure 3) usefully attributes part of the gain to temporal encoding.
• Real-robot section grounds the method. Even at $n=10$, the Sawyer results with 5–10 human labels demonstrate that the pipeline is not a simulator-only artefact and separate reward outputs on success vs. failure trajectories (Figures 7, 14) are illustrative.

Weaknesses

• Underspecified OT machinery. The $S_{\max}$ normalisation used to obtain $S_{\text{norm}}\in[-1,1]$ is defined informally as "the OT plan maximizes the contribution of all non-zero $R_{ij}$ terms," but those terms compete for the same marginal mass, so no closed form or proof is provided. Combined with the absence of an explicit map from sign(S_norm) to the $\tilde y\in\{0,1,0.5\}$ convention used by the cross-entropy loss (Eq. 1), and with the ternary matrix $R\in\{-1,0,1\}$ leaving equal-preference queries indistinguishable from "uncompared," the labelling pipeline is harder to reproduce than it should be.
• Effective size of the OT problem. With $M=2$ in Table 6, each Sinkhorn call couples the $N{=}20$ labeled segments to only $2$ unlabeled segments — one candidate pair at a time. That is much smaller than the "align visual trajectories" framing suggests and is never ablated against larger $M$, so a natural question — does OT do useful work beyond ranking candidate pairs by matched-labelled-pair alignment — is not answered.
• Ambiguous OT-vs-encoder attribution. The SIM-individual/mean/weighted baselines are defined only in prose and never state whether they share VOTP's S3D encoder. If they do not, Figure 2 conflates the OT contribution with the encoder contribution — precisely the central methodological question.
• Narrow empirical footprint framed broadly. The label-budget sweep (Figure 4) covers 4 tasks, the distractor robustness table covers 2, real-robot evaluation is $n=10$ per method, and Table 1 margins are frequently within one 5-seed standard deviation. Several conclusions ("high feedback efficiency," "robust to nuisance variation," "outperforms GT rewards on door-open") generalize beyond what these sample sizes support.
• Protocol seams around scripted vs. human labels. hopper-medium-replay-v2 silently switches to human labels because "scripted labels remain ineffective across baselines even when provided in large quantities." This is either a scope limitation of scripted teachers on that task (which affects the whole benchmark) or an unequal comparison, and the footnote does not say whether the 10-label budget still applies.
• Overreach in the "high-quality pseudo-labels" claim. Table 13's sweep-into row is 67.0% — only modestly above chance for a binary comparison — and Figure 17h shows the same task has the weakest reward correlation. The summary claim ignores this outlier.
• Statistical reporting is thin on the headline table. Table 1 reports 5-seed mean±std but no confidence intervals or paired-seed tests, while a 5% bolding rule masks overlap. Aggregate metrics with 95% CIs are shown in Figures 12/13 for a subset of methods — extending that to Table 1 would materially help the reader assess wins.
• Numerical inconsistencies. The main text says 50 teleoperated demonstrations per real-robot task while Appendix A.4 gives 40 for Lift Banana and 50 for Drawer Open; Table 12 lists identical 57.6±5.4 for plate-slide and sweep-into in the scripted-teacher column, which is statistically implausible and inconsistent with Table 1's per-task means.

Reproducibility & code

The paper documents the pipeline reasonably well — the S3D encoder, the POT Sinkhorn solver, uniform marginals with Euclidean cost, IQL/reward-model hyperparameters (Tables 4–7), the D4RL/MetaWorld preference-pair indices from Kim et al. (2023) and Hejna et al. (2024), and baseline URLs (Table 8). But several code-and-configuration levers that determine the reported numbers are missing:

• VOTP source and preprocessing not released. No repository is provided for VOTP itself, and neither the input pipeline for each encoder in Figure 3 (frame sampling rate, temporal window, per-frame preprocessing) nor the rendered visual segments used for pseudo-labeling are shipped. A third party would have to reconstruct these end-to-end.
• Sinkhorn regularization $\varepsilon$ is unspecified. $\varepsilon$ controls the sharpness of the transport plan and interacts with the cost function's magnitude (Euclidean vs. cosine); neither its value, iteration count, nor stopping criterion is stated, so the plans used to produce Eq. (5) cannot be recovered.
• Per-task $\tau_P$ is a small set, not a pinned value. Table 6 gives $\tau_P\in\{0.15,0.2\}$ (D4RL) and $\{0.35,0.4,0.45\}$ (MetaWorld), and the selection rule ("within the observed range of preference scores") is not tied to a validation signal. Figure 5 shows non-trivial sensitivity.
• Baseline re-tuning introduces hidden freedom. Table 1 depends on the authors' own hyperparameter search for eight competing methods with only sketchy grids in Table 8, no selection criterion, and no per-configuration variance.
• Distractor protocol underspecified. Table 2 references RL-ViGen but the lighting/texture ranges, the "easy/hard" background videos, and the compositing procedure are not given.
• Real-robot artefacts not released. The 40–50 teleoperated demonstrations per task, the 5/10 human preference labels, and the workspace calibration are not shipped, and the platform (Rethink Sawyer) is discontinued — making Table 3 impractical to reproduce independently.
• MetaWorld human-annotation protocol undocumented. Four annotators are mentioned in Section C.4 but no interface, instructions, quality control, inter-annotator agreement, or raw labels are provided.

None of these individually threaten the paper's central claims, but taken together they mean a third-party replication of Table 1 would have to guess several free parameters that plausibly move averages by several points.

Recommended Changes

Essential

• Formalise the pseudo-label pipeline. Write down a closed-form or explicit-bound argument for $S_{\max}$ that establishes $S_{\text{norm}}\in[-1,1]$ over the feasible-transport-plan polytope, and add a one-line equation for the map from sign(S_norm) (with $|S_{\text{norm}}|>\tau_P$) to the $\tilde y\in\{0,1,0.5\}$ label used in Eq. (1). State explicitly how equal-preference labels are handled in $R$.
• Isolate OT from the encoder. Confirm in text that the SIM-individual/mean/weighted baselines use the same S3D features and Euclidean distance as VOTP, provide equations for each variant, and — if not already the case — re-run them under the shared-encoder protocol. This is the ablation the paper's central claim rests on.
• Ablate $M$. Add results for $M\in\{2, 8, 32, 128\}$ (or similar) so that readers can see whether coupling more unlabeled segments per Sinkhorn call helps and whether the current $M=2$ choice is essential.
• Release code and per-task configurations. Ship (i) the VOTP repository, (ii) Sinkhorn $\varepsilon$ / iteration count, (iii) the pinned per-task $\tau_P$ value, (iv) the rendered visual segments, and (v) the exact encoder-preprocessing spec for R3M/CLIP/S3D/VideoCLIP/InternVideo in Figure 3.
• Strengthen the real-robot evaluation. Report more rollouts per method (e.g., $n\geq 30$), Wilson or bootstrap CIs on the success rates, and rollouts across multiple policy seeds so that Table 3's 80% vs 50% margin can be calibrated. Release the demonstration set, the human labels, and the workspace calibration details.

Suggested

• Add statistical framing to Table 1. Report bootstrap 95% CIs or paired-seed tests on the headline table (as is already done for the aggregate metrics in Figures 12–13) so that "within 5% of best" bolding is replaced by an interpretable significance criterion.
• Qualify the pseudo-label quality claim. In §C.5, explicitly note the sweep-into 67% row and connect it to the low reward correlation in Figure 17h; consider a short analysis of why S3D struggles on that task.
• Clarify the hopper-medium-replay footnote. State whether the 10-label budget applies to the human-labeled run, and discuss why scripted teachers are said to fail there — either as a scope limitation for scripted PbRL benchmarks or as a task-specific artefact.
• Broaden the robustness study. Extend Table 2 to plate-slide and sweep-into so the "robust to nuisance variation" claim is supported beyond two tasks.
• Fix numerical inconsistencies. Reconcile the 40 vs 50 demonstrations for Lift Banana between §5.5 and Appendix A.4, and audit Table 12's scripted-teacher row (identical 57.6±5.4 for plate-slide and sweep-into is likely a copy-paste error and inconsistent with Table 1).
• Reframe the uniformity justification. Rewrite Appendix C.8 so that uniform marginals are presented as a modelling choice with its consequences, rather than as a property forced by sampling-without-replacement.
• Discuss the door-open above-GT result. Add a short note distinguishing seed noise from reward-shaping smoothing as candidate mechanisms for VOTP's 84.0 vs IQL+GT's 79.2 on door-open, so readers do not over-interpret the result.
• Complete the cost-metric ablation. Add hopper-medium-expert-v2 and walker2d-medium-expert-v2 rows to Table 9, or state explicitly why they are omitted.