CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Subtitle Removal

Summary

CLEAR proposes a mask-free video subtitle removal framework built on a two-stage pipeline. Stage I extracts a coarse occlusion prior from paired subtitled/clean videos via self-supervised disentanglement — dual encoders with orthogonality and adversarial losses over pixel-difference pseudo-labels. Stage II attaches a LoRA to a frozen video diffusion transformer plus a lightweight occlusion head that predicts an adaptive weight map from the DiT's mid encoder features; a joint objective of distillation, generation-feedback, and sparsity/KL terms is meant to let this map self-correct without ground-truth masks. At inference the head runs internally and no mask or text detector is required. On a self-collected 400-sample Chinese benchmark CLEAR reports +6.77 dB PSNR and −74.7% VFID over three mask-guided baselines, plus qualitative zero-shot transfer to six languages, while training a small fraction of the base model.

The conceptual move — turning the mask from an inference-time input into an internal, generation-supervised weighting map — is a useful reframing for a task where per-frame annotation is expensive and detector drift is a real failure mode. The two-stage decoupling and small trainable footprint are attractive engineering choices, and the qualitative behavior holds up under execution: a single Chinese-trained checkpoint removes subtitles cleanly across Latin, Arabic (RTL), and CJK scripts with no language-specific tuning, and the mask-free inference path runs as shipped at a competitive (indeed faster) per-frame speed.

However, the quantitative story is unconfirmed and one efficiency claim is contradicted by the released artifact. The self-correction gradient derived from the stated objective has the opposite sign to the text; the base-model size, LoRA rank, and parameter fraction are mutually inconsistent; and the −74.7% VFID figure matches no Table 1 baseline. On execution the released checkpoint is a rank-64 LoRA whose trainable ratio is ~6.31%, roughly 8x the headline 0.77%. Every reconstruction/perceptual/temporal number rests on a single private 400-sample Chinese benchmark with no error bars, no metrics code shipped, and baselines handed oracle GT-derived masks, so none of Tables 1–3 could be recomputed.

Strengths

• Conceptual reframing. Internalizing the adaptive mask as a weighting map supervised by generation feedback removes an inference-time dependency (external text detection or per-frame annotation) that has been a genuine bottleneck for mask-guided video subtitle removers.
• Parameter-efficient adaptation. Decoupling a coarse self-supervised prior from LoRA adaptation of a frozen video diffusion prior plus a small occlusion head is a reasonable way to keep the trainable footprint small and preserve pretrained visual priors.
• Mask-free inference reproduces and runs cleanly. The shipped inference path ran with the released checkpoint and no source edits, producing clean subtitle removal; measured speed (~3.0 s/frame, 1.76 s/frame contention-free) is the same order as — and faster than — the reported 4.86 s/frame.
• Cross-lingual behavior is real, not just claimed. A single Chinese-trained checkpoint removed subtitles zero-shot on English, Arabic, and Japanese demo clips, spanning three distinct scripts including RTL and CJK, with background texture preserved.
• Consistent multi-metric margins (as reported). On the paper's benchmark the reported gaps over ProPainter, MiniMax-Remover, and DiffuEraser are large across reconstruction, perceptual, and temporal metrics simultaneously, which is harder to attribute to a single metric quirk — though these numbers remain unverified here.

Weaknesses

• Self-correction mechanism contradicted by its own loss form. For $w = (1 + \alpha(k)\cdot M^{pred})(\epsilon^{gen}+\delta)^\gamma \cdot L_{diff}$, $\partial L / \partial M^{pred} \approx \alpha(k)(\epsilon^{gen}+\delta)^\gamma L_{diff} \ge 0$, so gradient descent decreases $M^{pred}$ where $\epsilon^{gen}$ is large — the opposite of the claimed self-correction. The asserted $O(\epsilon^{gen})$ chain-rule terms are never derived. This is the central Stage-II mechanism.
• Efficiency accounting is internally inconsistent and contradicted by the artifact. The abstract's 0.77% and Figure 3's ~112M params (0.8% of 14B) are consistent only with a 14B base, but Section 4.2 states a 1.3B base; the LoRA rank appears as both $r=8$ and $r=64$. Execution shows the shipped checkpoint is rank-64 with a ~6.31% trainable ratio — ~8x the headline (see Reproducibility & code).
• Headline VFID reduction matches no baseline. −74.7% follows from none of the Table 1 baselines (arithmetic gives 71.9%, 78.6%, or 79.3%).
• Underspecified inference/definitions. Algorithm 1 initializes $z^{sub}_0$ but never consumes it; the difficulty error decodes the fixed clean latent $z_0$ rather than a predicted $\hat{x}_0$; $\epsilon^{gen}$ is squared $L_2$ while Stage I uses plain $L_2$ (changing the effective focal exponent from 0.8 to 1.6).
• Baselines receive oracle GT-derived masks. Masks from $\|X^{sub}-X^{clean}\|_2$ use the ground-truth clean video unavailable at deployment and are a coarse stand-in for the scene-text detectors these baselines were designed for; a detector-mask comparison is missing.
• Cross-lingual generalization is not quantified. The six-language claim rests on qualitative panels — no per-language metrics and no held-out non-Chinese benchmark, despite this being measurable.
• Thin statistical support and a weak ablation baseline. All numbers come from one 400-sample benchmark with no seeds/CIs; per-module deltas (M3 +0.37 dB) are within plausible LoRA seed noise, and the LoRA-only baseline is already fine-tuned, so no row isolates the value of any adaptation from the CLEAR-specific modules.
• Advertised vs used data. Section 4.2 announces over 160,000 pairs but Stage I and Stage II each use 500.
• Consistency/labelling issues. all attention layers vs the (q,k,v,o,ffn.0,ffn.2) target set; middle layer vs layer 18/42; $T_{period}=30$ vs $40$; 2D head equation vs 3D-conv claim; undefined random blackout augmentation; Figures A2–A4 captions list Arabic for German; the cfg discussion claims cfg=1.0 wins on fidelity though cfg=1.2 has the higher PSNR.

Reproducibility & code

(veritas executed the shipped inference path with the released checkpoint; the Stage-II training path and all metrics were not runnable from the released materials.)

• Parameter-efficiency claim fails against the released checkpoint. Counting tensors directly, the shipped LoRA is rank-64 (600 tensors, 87.49M params over q,k,v,o,ffn.0,ffn.2 in all 30 blocks) plus a ~2.06M head, a ~6.31% trainable ratio versus the claimed 0.77% (~8x off). 0.77% is consistent only with a rank-8 LoRA; the artifact is rank-64 (the repo default). This is a paper/artifact discrepancy, not a run defect.
• Headline metrics unreachable. No PSNR/SSIM/LPIPS/DISTS/VFID/TWE/Flow implementation ships, no VFID/Inception-3D dependency, no baseline runners, and the 400-sample benchmark is private with no download path. A from-scratch metrics harness was written and math-validated but produced num_pairs=0 for lack of paired ground truth, so Table 1, the metric side of Tables 2–3, the +6.77 dB PSNR gap, the −74.7% VFID, and −93.0% Flow Var could not be checked; no values were invented.
• Inference and cross-lingual behavior reproduce. The mask-free pipeline ran as shipped; measured speed ~3.0 s/frame (faster than 4.86); subtitle removal confirmed on English, Arabic, and Japanese clips. Only 2 of the 6 named languages were testable — Korean, French, Russian, and German ship no demo clips (Arabic, which is shipped, is not among the six).
• Timing axis of Table 3 diverges. At default lora_scale, steps 5→10 raised per-frame time only 1.07x (1.93→2.07 s), not the claimed ~2x, because the run used a downsized 81-frame clip where fixed VAE/IO cost dominates; other Table 3 columns were data-blocked.
• Stage-II training path not exercised. The occlusion head and the M2/M3/M4 losses were not run (no training executed in the reproduction), and paper-only code inspection reported the head is imported but not clearly wired into the training step; the released grid-search script also assumes unshipped training checkpoints. So Table 2's progressive ablation is not reproducible end-to-end.

Recommended Changes

Essential

• Reconcile the efficiency claim with the released checkpoint. State the base-model size (1.3B), the single LoRA rank actually used (the artifact is rank-64), and the true trainable parameter count and fraction (~6.31%), so the abstract, Figure 3 (112M / 0.8% of 14B), and Section 4.2 agree; if 0.77% refers to a rank-8 configuration, ship that checkpoint.
• Derive the self-correction gradient explicitly. Show that $\partial L_{gen}/\partial M^{pred}$ increases $M^{pred}$ where $\epsilon^{gen}$ is large, or spell out which term (distillation/consistency, a sigmoid, or a clip) reverses the sign; a numerical log of $\Delta M^{pred}$ vs $\epsilon^{gen}$ during training would settle it.
• Release the metric-evaluation code and a paired benchmark. Ship PSNR/SSIM/LPIPS/DISTS/VFID/TWE/Flow scripts (with the VFID/DISTS feature models) plus a hosted paired sub/+clean/ set so Tables 1–3 are reproducible; document that these are required, since a reproducer currently must reimplement all metrics from scratch.
• Correct or source the −74.7% VFID number. State the baseline and arithmetic that yield it, or fix it.
• Add a quantitative cross-lingual evaluation on at least one non-Chinese paired set, and correct the Arabic↔German captions in Figures A2–A4.

Suggested

• Report per-seed variance for the ablation. With M3 at +0.37 dB, give a few seeds per configuration so signal is separable from noise.
• Add a no-adaptation baseline row (frozen Wan2.1-Fun-V1.1-1.3B, prompt only) to Table 2 to isolate the CLEAR modules from the LoRA fine-tune.
• Add a detector-mask baseline configuration. Run ProPainter/MiniMax-Remover/DiffuEraser with a scene-text detector on $X^{sub}$ alongside the oracle GT-difference mask.
• Specify the inference conditioning path in Algorithm 1 (the control-latent input for $X^{sub}$), and disambiguate $\hat{x}=\text{VAEdec}(z_0)$ (predicted $\hat{x}_0$ vs clean $z_0$) and the $\epsilon^{gen}$ norm convention.
• Pin down architecture/hyperparameters: exact tap layer, $T_{period}$, 2D vs 3D head kernel, the full LoRA target set vs all attention layers, and the random blackout augmentation schedule.
• Reconcile the 160,000-pair headline with the 500-pair training runs, ship demo clips for the four missing languages, and rephrase the cfg discussion to acknowledge cfg=1.2 wins on PSNR.