Motion Attribution for Video Generation

Summary

MOTIVE proposes a motion-aware data-attribution framework for text-to-video diffusion / flow-matching models. The core conceptual move is to reweight per-location denoising errors by an optical-flow-derived motion mask before taking gradients, so that the resulting influence scores emphasize training clips that shape temporal dynamics rather than static appearance. The paper couples this idea with three scalability choices — a single-timestep estimator with common randomness, a Fastfood Johnson-Lindenstrauss projection to $D' = 512$, and a per-video frame-length correction — that together bring gradient-based attribution to the scale of Wan2.1-T2V-1.3B, Wan2.2-TI2V-5B, and LTX-2B on 10k-clip subsets of VIDGEN-1M and 4DNeX-10M. The framework is used to score 10k clips against 50 Veo-3-generated motion queries (ten categories, majority-vote aggregation), select the top 10%, and fine-tune. On VBench, MOTIVE improves Dynamic Degree substantially over Random, Motion-Magnitude, V-JEPA, and its own no-motion-mask ablation across all three backbones; humans prefer MOTIVE-fine-tuned outputs 74.1% of the time against the pretrained base. This matters because motion is what distinguishes video generation from image generation, and prior diffusion-attribution work has been limited to appearance in still images, so a motion-centric influence signal is a genuine gap-filler and a plausible foundation for data curation. Overall the contribution is real: the motion-mask formulation is clean, the empirical setup is realistic, and Dynamic-Degree gains at 10% data are noticeable. That said, the main conceptual limitation is that the influence estimator sits far below the theoretical target — cosine similarity of unit-normalized projected gradients at a single fixed $(t, \epsilon)$ — and each layer of approximation is validated only weakly (rho ~ 66% at $t = 751$ against a 10-timestep baseline; rho ~ 75% at $D' = 512$ against full gradients). The empirical evaluation also compares only to selection heuristics, not to any diffusion-attribution method from the cited literature, so the marginal value of attribution over motion-aware sampling is not fully established.

Strengths

• Conceptual contribution. A clean, well-motivated formulation: reweight the per-location squared error by a normalized motion mask so gradient-based influence is dominated by dynamic regions rather than appearance. This turns the largely image-focused attribution literature into a motion-first tool without touching the generator's training objective (§3.4).
• Scalability engineering. The combination of single-timestep + common randomness + Fastfood projection + per-video frame-length normalization is what makes attribution feasible at Wan-scale, and the runtime breakdown (Tab. 6) makes the trade-offs concrete (150 h one-time, 54 s per new query).
• Backbone and dataset coverage. The evaluation spans two model families (Wan2.1 / Wan2.2 and LTX-2B) and two datasets (VIDGEN-1M, 4DNeX-10M), and results are consistent enough across settings to suggest the phenomenon is real and not a single-backbone artifact.
• Motion is not just motion-magnitude. Figure 6's overlap of top-10% and bottom-10% motion-magnitude distributions, together with the cross-motion overlap heatmaps in Figure 7, argues persuasively that MOTIVE's rankings are not a proxy for a saliency-only filter, and the fact that both datasets reproduce the same high- and low-overlap pairs (bounce-float, free-fall-stretch) supports a data-independent motion structure.
• Human evaluation supplement. VBench alone would be thin; adding a 17-annotator pairwise study broadens the evidence for perceptual gains, and Table 2 also shows the (revealing) small margin against Full FT (53.1% win) which the authors do not oversell.

Weaknesses

• Estimator validation is weak. The single-timestep design agrees with a 10-timestep baseline at only rho=66% (Sec. 4.4), and the D'=512 projection agrees with the full-gradient ranking at only rho ~ 74.7% (ceiling rho ~ 76.1% at D'=2048). Both are framed as 'closely matching' / 'maintaining quality' but represent moderate rank noise, and both compound because top-10% cutoffs are precisely where boundary reshuffling matters. Top-K set-overlap (not rank correlation) is the quantity that actually controls the fine-tuning outcome; it is never reported.
• Timestep-bias argument versus single-sample estimator. Section 2.2 warns about timestep-scale bias in diffusion gradients, then §3.2 collapses to a single fixed $t_{\text{fix}} = 751$ and cites DiffusionReTrac / Diffusion Attribution Score to justify it. Those references motivate sub-sampling and normalizing across $t$, not single-sampling; the paper does not show that the systematic bias at $t = 751$ is small, nor does it explain why 751 is called the 'midpoint of the 1000-step trajectory' when the arithmetic midpoint is 500.
• Frame-length correction interacts oddly with L2 normalization. Because Eq. 10 unit-normalizes the projected gradient and Eq. 17 is a cosine similarity, any per-example scalar (like the $1/F$ correction in Algorithm 1 step 6, or the $1/F_v$ factor in the motion loss) drops out of the final influence score. Combined with the fact that clips are standardized to 81 frames (F.1), the $/F$ correction cannot on its own change the ranking, so the mechanism by which frame-length bias is 'fixed' is unclear. The Sec. 4.4 ablation compares standardized vs unstandardized clips, not with vs without the $/F$ correction, so it does not isolate the claimed contribution of Eq. 11.
• Missing attribution baseline. No prior gradient-based diffusion attribution method (DiffusionReTrac, TRAK-for-diffusion, Diffusion Attribution Score, Concept-TRAK) is run on the same task. 'Ours w/o motion masking' is only an ablation of MOTIVE's own pipeline. Without a comparator from the cited literature, the paper cannot separate the value of motion masking specifically from the value of a scalable gradient-similarity pipeline in general.
• Statistical support for headline numbers. Table 1 reports single-run point estimates without seed averaging, variance, or significance tests, yet the text says MOTIVE 'significantly outperforms' the baselines. Given known variance in VBench sampling and fine-tuning, several of the reported margins (especially against Ours w/o MM and Full FT) may not survive a proper repeated-run treatment.
• Trade-off with non-motion metrics is downplayed. Table 1 shows Imaging Quality regressions relative to base (65.7 -> 64.6 on Wan2.1) and relative to Full FT on Wan2.2 (66.2 -> 65.5), plus Background Consistency drops relative to Full FT (96.6 -> 96.1). The abstract claims MOTIVE 'matches or surpasses' full fine-tuning; that is true for Dynamic Degree, not uniformly across VBench dimensions. This trade-off deserves more than one sentence of Limitations.
• Synthetic query set. All 50 queries are Veo-3-generated. This gives clean stimuli, but it also means both the attribution signal and the human evaluation reference are shaped by a specific model's understanding of each motion. The authors do not sanity-check with real-video queries.
• Camera-only motion handling is under-specified. The Limitations says clips with 'spatial uniformity of W' are 'down-weighted', which sounds like a live component of the method, but no threshold or down-weighting rule is provided. The reader cannot reproduce or ablate this behavior.
• Human evaluation arithmetic and design. The '850 total' figure in Table 2's caption is consistent with 17 raters $\times$ 50 test cases, but §4.3 also says each category involves 'three pairings' (and Table 2 shows four comparison rows). Under a fully-crossed design this would imply 17 $\times$ 150 or 17 $\times$ 200 judgments. The allocation of comparisons across raters and pairings is not described.
• Minor numerical / notation issues. The compress-float overlap number (40.1% for 4DNEX) does not match Figure 7. The motion field is denoted $\mathbf{F}_f$ in §2.1 and redefined as $\mathbf{D}_f$ in §3.4 for the same object, while $F$ is also the frame count. The min-max normalization uses $M_f(h', w')$ under a min/max over $f', h', w'$ — the subscript should be $f'$. Tab. 6 and Tab. 7 are cross-referenced interchangeably for the 150-h figure. Individually small, but together they slow careful reading.

Reproducibility & code

Code was inspected only through what the manuscript describes (paper-code mode); no released implementation was surfaced for hands-on inspection. On that basis:

• No code/checkpoint/query-set release surfaced. Only a project page is listed. The end-to-end pipeline (Fastfood projection with seed, DiffSynth-Studio fine-tuning wrapper, camera-motion detector, majority-vote threshold $\tau$, VBench harness) would need to be reconstructed from prose, and neither the 50 Veo-3 queries nor the 10k VIDGEN / 10k 4DNeX subsets are released.
• Seed protocol is not stated. Fastfood projection matrix, single-sample $\epsilon_{\text{fix}}$, fine-tuning shuffling / init, and VBench sampling are all sources of run-to-run variance that would move Table 1 numbers by non-trivial amounts. No seeds are pinned.
• Fine-tuning hyperparameters are incomplete. Appendix F.1 covers resolution, learning rate, and epoch count, but omits batch size, gradient accumulation, warm-up, gradient clipping, precision, and what '1 epoch, dataset repeated 50 times' means concretely. These matter for a ~1000-clip top-10% subset.
• VBench sampling configuration missing. Samples per prompt, sampler choice, guidance scale, and prompt list are unstated — all levers that VBench numbers depend on, especially at the 2-5 pp margin visible in Table 1.
• Human-evaluation harness undocumented. No annotator instructions, no rater recruitment source, no interface details, and no rule for how the 850 judgments were allocated across 17 raters $\times$ 4 comparison conditions.
• Baseline implementations underspecified. Motion Magnitude, Optical Flow, and V-JEPA baselines lack architecture-level and hyperparameter-level detail. Since these are the reference points against which MOTIVE's gains are measured, small choices could shift them.
• Full-gradient reference for the projection ablation. Fig. 5's rho values are computed against a 'full-gradient baseline' at 1.4B-parameter scale, but the procedure for actually storing and computing that baseline (which clips, which queries, what precision) is not described.

Recommended Changes

Essential

• Report top-K set-overlap alongside Spearman rho for the projection and single-timestep ablations. Section 4.4's rho=66% (single timestep) and rho=74.7% ($D' = 512$) do not directly speak to which clips actually enter the top-10% cutoff — the quantity that drives the fine-tuning gains. Add top-10% (and top-1%) Jaccard overlap between approximated and full/multi-timestep rankings, since 'maintaining quality' is claimed for a value that leaves substantial reordering.
• Add at least one prior diffusion-attribution baseline. Run one of DiffusionReTrac, Diffusion Attribution Score, or Concept-TRAK on the same 10k subsets and the same 50 queries, and report VBench + Table 1 columns. Without a diffusion-attribution comparator, the paper cannot isolate the value of motion masking from the value of gradient-similarity selection over motion-magnitude / V-JEPA / random selection.
• Report per-seed variance for Tables 1, 2, and 5. At minimum run three fine-tunes per row and give standard deviations, so that 'significantly outperforming' becomes a defensible claim rather than a rhetorical one. This is particularly important for the tight Ours vs. Ours w/o MM and Ours vs. Full FT gaps.
• Fix the frame-length-correction derivation and ablation. Show explicitly where the $1/F$ (or $1/F_v$) factor enters the final influence score once L2 normalization and cosine similarity are applied, and, if it does not, remove the redundant step. Re-do the Sec. 4.4 ablation to isolate the $/F$ correction from the clip-length standardization protocol.
• Correct the TRAK preconditioner citation. TRAK does not use an identity preconditioner; rephrase the §3.2 sentence to attribute the identity preconditioner to Koh & Liang / Pruthi et al. only, and describe TRAK's projected-Gauss-Newton preconditioner separately (or drop it here).
• Release the query set, 10k subsets, seeds, fine-tuning hyperparameters, and human-eval interface. Concretely: (i) the 50 Veo-3 query clips (or their generation seeds and prompts if licensing forbids the videos), (ii) file lists for the 10k VIDGEN and 10k 4DNeX subsets used, (iii) Fastfood seed and single-sample $\epsilon_{\text{fix}}$ seed, (iv) batch size, gradient accumulation, majority-vote percentile $\tau$, and (v) rater instructions and interface for the human study. Without these the numerical claims cannot be independently rechecked.

Suggested

• Sanity-check with real-video queries. Rerun the ranking on 10 real motion clips (one per category), report the top-10 selected training clips, and compare against Veo-3-query rankings. This isolates whether MOTIVE is generalizing motion or fitting Veo-3-specific stylizations.
• Add an appearance-quality panel to the main results. Foreground the Imaging-Quality and Background-Consistency regressions relative to base and Full FT so the appearance / motion trade-off is visible, and revise the abstract's 'matching, or surpassing, full-dataset fine-tuning performance' to explicitly say 'on Dynamic Degree'.
• Specify the camera-motion detector. State the uniformity statistic on $\mathbf{W}$, the threshold, and the down-weighting factor. Ideally add an ablation showing what fraction of clips are down-weighted and how much the fix changes VBench.
• Rebuild the human-evaluation arithmetic. Reconcile the '17 annotators, 5 test cases per category, 10 categories, three pairings, 850 total' description with the four comparison rows in Table 2, and specify how comparisons were assigned to raters.
• Fix the numerical / notation nits. Correct compress-float 40.1% to match Figure 7; unify $\mathbf{F}_f$ and $\mathbf{D}_f$ (and use a non-overloaded symbol for the frame count); primed subscript in the min-max normalization; and clarify Tab. 6 vs Tab. 7 references in §G.1.
• Clarify what 'closely matching' and 'midpoint' mean for $t = 751$. Either report the schedule that makes 751 a midpoint, or drop 'midpoint'; and either replace 'closely matching' with the rho value directly or add a top-K overlap that supports the phrase.