Multimodal Nested Learning for Decoupled and Coordinated Optimization

Summary

The paper attacks the well-documented problem of imbalanced multimodal optimization — strong modalities dominating gradient dynamics and suppressing informative weaker ones — from a fresh angle: instead of modulating gradients or resolving conflicts inside a monolithic joint-training loop, it reinterprets the pipeline as a two-level nested-learning (NL) system. The outer level, DMSM, treats each modality encoder and the joint classifier as independent associative memories with their own losses and their own momentum-SGD optimizers, with a modality-masking trick that turns the shared classifier into a per-modality readout. The inner level, AMCF, is a three-layer MLP cascade (CCMS) whose parameter-side error signal is EMA-decomposed into three temporal bands (Swift, Consensus, Steady) and routed to CCMS layers running at two update frequencies. The move is conceptually clean: it uses the NL vocabulary of associative memory at an update frequency to argue that decoupling the objectives and the update rates is what breaks the strong-modality feedback loop, and it hangs a plausible early-vs-late-training story on top of the EMA decomposition.

Empirically MoNet is the top fusion method on all eight datasets across three tasks, the ablations show sizeable drops when the outer or inner nested block is removed, and the training-curve and gradient-magnitude plots argue the mechanism does rebalance encoder learning. Executing the authors' released CREMA-D checkpoint through the shipped code reproduces the paper's headline audio-visual numbers to within about a point ($82.80$ vs. $83.11$ fusion), which corroborates the architecture on one dataset even if it does not independently reproduce training. The main conceptual limitations are that several load-bearing steps are asserted rather than derived — the identification of the Swift signal with the strong modality, the counter-intuitive routing of the fast-varying signal to the slowest layer, the elimination' of interference, and the higher-order temporal dependency' claim for DeepMomentum — and that `multi-timescale' is oversold: CCMS has three layers but only two frequencies. Missing equations and symbol overloads make the formal specification hard to reconstruct, and outside CREMA-D the release does not support end-to-end reproduction: the ten comparison methods are not implemented, the six ablation switches are not exposed, and none of the image-text or 2D-3D datasets is staged.

Strengths

• Conceptual contribution. Reframing the multimodal-imbalance problem as decoupled associative memories at different update frequencies is a genuinely different move from prior gradient-modulation and conflict-alleviation lines. The DMSM half of the design — independent optimizers over encoder and classifier objectives, coupled through a mask that turns the classifier into a per-modality readout — is clean and gives an intuitive reason why the strong-modality feedback loop is broken.
• Empirical breadth. Eight datasets across three modality pairs (audio–visual, image–text, 2D–3D) with $10$ baselines and three seeds is a serious test bed, and MoNet is the highest-scoring fusion method on all eight. The gains are largest on the weak modality (e.g., video on CREMA-D and AVSBench, image on FOOD101), which is where the mechanism is supposed to help.
• Diagnostic experiments. The training-curve overlays (Figures 4/7), the per-CCMS-layer influence histograms (Figure 5), the unimodal-contribution comparisons (Figures 1(a)/8), and the gradient-magnitude time series (Figure 9) triangulate the mechanism rather than only reporting final accuracies. Even where they overstate emergence, they make the story testable.
• Sensitivity range. Ablations show that the nested structure and DMSM in particular carry most of the gains ($\Delta = 19.85$ and $51.97$ pp respectively on CREMA-D), and the $\beta_1$/$\beta_2$ sensitivity study shows a wide insensitive band, arguing the EMA decomposition is not brittle to hyperparameter choice.
• Artifact-level match on the headline number. Loading the released CREMA-D checkpoint (fyl8158/MoNet) strictly into the shipped AVClassifier_NL and re-evaluating on the $744$-sample test split reproduces fusion $82.80\%$, audio $61.56\%$, video $73.66\%$ — within $0.3$–$1.7$ pp of Table 1. That is genuine corroboration that the released architecture and weights are the ones behind the paper's headline audio-visual number.

Weaknesses

• Counter-intuitive routing is asserted, not justified. The Swift signal (rapidly varying, $\delta - \delta_s$) is routed to the slowest layer while the Steady signal ($\delta_l$, slow-accumulating) is routed to a fast layer. The stated rationale (slow down dominant modality signals' / accelerate memory integration') is intuitive but not derived, and a symmetric argument could support the opposite mapping. The whole rebalancing story leans on this choice.
• Only two update frequencies for a three-layer CCMS. Section 3.3 sends the Consensus and Steady signals to layers at the same $f_f = \gamma f_s$, so `multi-timescale with three layers' is an overstatement. Nothing else in the description distinguishes the two fast layers.
• Load-bearing identifications made without evidence. The paper's mechanism depends on identifying the Swift signal with the strong modality (Section 3.3), yet this is asserted with a hedge (typically induced by strong modalities') and never demonstrated. Similarly, the adaptive assignment' in Figure 5 is largely a consequence of the fixed routing, not an emergent property.
• Overclaimed guarantees. Effectively eliminating interference' (Section 3.2) and higher-order temporal dependencies' (Appendix A.5) both promise strictly stronger properties than the mechanism delivers — decoupled optimizers still communicate through the shared fused feature and $W_c$, and a stack of first-order EMAs is not a higher-order dependency. These are load-bearing framings in the conclusion.
• Unimodal-performance claim is uneven. The statement that MoNet delivers `enhanced unimodal performance' is only partially true — MoNet's AUDIO is lower than MMPareto/DGL on CREMA-D and lower than MMCosine/MLA on AVSBench and VGGSound50. The accurate and more informative claim is that MoNet raises the weak modality without collapsing the strong one.
• Under-specified formalism. Several central equations (encoder/classifier objectives, $\delta$, the decomposition, and the DeepMomentum update) are announced but not rendered. Meanwhile $\lambda$ is used as both an outer-loss weight ($\lambda = 10$) and a frequency ratio ($f_s = f_f/\lambda$) in the appendix, and $f_A$/$f_B$ appear without being tied back to $f_f$/$f_s$. `Deviation of instantaneous gradients' in Section 3.4 is never defined.
• Interpretation of $w/o$ AMCF ablation. Table 3 shows $w/o$ AMCF lowers Video ($74.33 \rightarrow 72.96$), not raises it, contradicting the paper's blanket statement that `unimodal performance may slightly improve'.
• Dataset bookkeeping errors. CREMA-D prose says $7{,}442$ total clips but Table 4 lists $6{,}697 / 744 / 744$ ($= 8{,}185$). MVSA prose says $518$ val / $519$ test while Table 4 says $519 / 518$. AVSBench has no citation and no version pointer. These are small but exactly the details a reproducer needs.
• Algorithm 1 ordering. As written, encoders are optimized (step 6) before the fusion forward that would produce their loss (step 7); either a forward pass is missing or the update uses stale gradients.
• Relative-influence metric under-specified. Neither the norm convention nor any normalization is stated, and $\delta_a, \delta_v$ are not defined precisely enough to reproduce the histograms in Figure 5.
• Strong-modality identity is not stable. The paper presents audio as the strong modality on CREMA-D, but under OGM-GE and under MoNet's own trained model video is stronger than audio; the mechanism's story should say how it behaves once that ranking flips.

Reproducibility & code

The paper's method code runs as shipped. That is worth stating up front: the released main_nl_av.py / basic_model.py / AVdataset.py needed no method-level patches to produce results, and evaluating the released CREMA-D checkpoint on the shipped test loader reaches fusion $82.80\%$ — $0.31$ pp under the paper's $83.11\%$ — with a strict state-dict load. Beyond that one artifact-level match, however, the release does not support end-to-end reproduction of most of the paper's evidence.

• Headline table (CREMA-D). The single MoNet cell that reproduced comes from evaluating the authors' released weights; from-scratch training on a $1$-CPU cpuset was truncated at $20$ of $100$ epochs and reached only $\sim67\%$ fusion. That is a compute-budget artifact, not a refutation, but it means the run does not independently reproduce the training procedure. CREMA-D also has no held-out val split — the code falls back to val == test, so the released best' checkpoint is itself test-selected, an undisclosed protocol note. The released README reports $83.33\%$ while re-eval yields $82.80\%$, likely traceable to unsorted os.listdir` frame indexing that makes the pipeline filesystem-order dependent.
• Comparison tables (all eight datasets, ten methods). models/fusion_modules.py implements only Sum/Concat/FiLM/Gated; the --modulation CLI flag is parsed but never read, so selecting OGM_GE silently runs an unmodulated model. None of OGM-GE, AGM, MMCosine, MMPareto, MLA, DGL, ARL, or InfoReg can be run from this repo, and Tables 1–2 and Figures 1(b), 4, 7, 8 cannot be reproduced end-to-end without re-implementing eight external baselines.
• Image-text and 2D-3D halves of Table 2. Never attempted by the replication: MVSA, FOOD101, 3D MNIST, and ModelNet raw data were not staged. Two parts are the paper's fault: the 3D MNIST Dropbox link ships only precomputed features while dataset/Mnistdataset.py requires Kaggle-gated raw *_point_clouds.h5, and the README cites cramed_nl_vl.sh, which does not exist (the actual image-text script, cramed_dgl_vl.sh, launches FOOD101 only, not MVSA).
• Ablation table (Table 3). No CLI switches or config keys expose the six variants; reproducing them requires hand-editing multiple source files with no documented label-to-code mapping. When the replication implemented the switches and ran all seven variants at a matched $8$-epoch budget, the $w/o$ DMSM row remained the largest drop directionally ($23.25$ pp vs. paper $51.97$), but the whole-nested-learning ablation ($w/o$ NL) went from a $-19.85$ pp drop in the paper to a $+0.13$ pp tie in the replication, and the ordering of the inner-level drops (CCMS/EMA/$f_f = f_s = 1$/AMCF) does not match the paper's ordering. Compute truncation is the most plausible cause, but the label-to-code mapping needs to be documented so this can be verified independently.
• Sensitivity sweeps (Figures 3 and 6). $\gamma$, $\beta_1$, $\beta_2$ are hard-coded in models/basic_model.py; $\lambda$ has three conflicting values in the release (paper: $10$; cramed_nl_av.sh: --alpha 15; argparse default: $4.0$ with help text reading alpha in DGL). When the replication ran a matched $8$-epoch sweep, the fusion accuracy varied $10.9$ pp across $\lambda \in \{1,5,10,15\}$ and rose monotonically to $\lambda = 15$ rather than peaking at $10$; the $\gamma$ optimum did land at $\gamma = 2$. The `flatness' the paper reports is not reproduced under this budget.
• Determinism. setup_seed is defined in utils/utils.py but is not called from any main_nl_*.py entry point, so the mean $\pm$ std over three random runs reported in the tables cannot be reproduced exactly without patches. main_nl_vl.py also references args.random_seed at the best-checkpoint save site without ever adding it to argparse, which will crash MVSA/FOOD101 runs before the first checkpoint is written.
• VGGSound50 validation = test. For VGGSound50 the code sets the validation loader equal to the test loader (no held-out val split); model selection therefore reads from the test set on VGGSound50 — an undisclosed protocol note that matters for the $60.90\%$ headline.
• Analysis infrastructure. The unimodal-contribution measure $s^j$ (Appendix A.1) is not implemented; per-batch encoder gradient magnitudes (Figure 9) are not logged in train_epoch; per-layer CCMS error signals (Figure 5) are appended to in-memory lists but never persisted, stage-tagged, or paired with $\delta_a$/$\delta_v$. When the replication reconstructed Figure 1(a) from normalized unimodal test accuracy (rather than $s^j$) it found MoNet and Concatenation with nearly identical audio/video balance — most likely a metric-substitution artifact, but one a reader cannot check without shipped code.
• Artifact identity. The linked repository's README describes the work as Information-Balanced Multimodal Learning (IBML) with nested BIO optimization and TCM task-complexity modulation — none of which is the paper's vocabulary (MoNet / DMSM / AMCF / CCMS). The Python code does implement MoNet as described, but the mismatched README (apparently carried over from a companion paper) and the checkpoint-vs-paper number gap should be reconciled so a reader can be confident the released artifact is the one the paper measured.

Recommended Changes

Essential

• Justify — or empirically test — the Swift-vs-strong-modality identification and the fast-signal-to-slow-layer routing. Address the two load-bearing assertions flagged in Counter-intuitive routing' and Load-bearing identifications' under Weaknesses. A per-signal ablation that swaps or shuffles which layer receives Swift/Consensus/Steady, plus a diagnostic showing the temporal correlation between $\delta - \delta_s$ and per-modality error signals, would move the CCMS story from asserted to established.
• **Reconcile multi-timescale' with only two frequencies.** Either introduce a third distinct frequency for the third CCMS layer or restate CCMS honestly as a two-timescale system with three signal inputs (Only two update frequencies' in Weaknesses).
• Soften the overclaimed statements. Replace effectively eliminating interference' with reducing/mitigating'; drop or substantiate higher-order temporal dependencies' (a stack of first-order EMAs is not one); rewrite delivers enhanced unimodal performance' to describe accurately the weak-modality lift without the strong-modality drop shown by Tables 1–2.
• Render the missing equations. The DMSM encoder objective, the classifier objective, Equation (5) for $\delta$, Equation (6) for the decomposition, and the DeepMomentum update all appear as can be formulated as:' without the formula. Also define deviation of instantaneous gradients' explicitly (`Under-specified formalism' in Weaknesses).
• Disambiguate $\lambda$ and align notation. Use a distinct symbol for the frequency ratio in Appendix A.4 (or reuse $\gamma$), and tie the $f_A$/$f_B$ ordering back to $f_f$/$f_s$.
• Ship the ten baselines and the six Table 3 ablations as first-class options. Either implement (or wrap the authors' released versions of) OGM-GE / AGM / MMCosine / MMPareto / MLA / DGL / ARL / InfoReg, and expose the six Table 3 ablations as CLI switches with a documented label-to-code map. As things stand, the --modulation flag is parsed but never read, silently returning an unmodulated run, which is worse than an outright omission.
• Investigate and disclose the $w/o$ NL ablation gap. The paper reports a $19.85$ pp drop; the replication at a matched budget shows a $0.13$ pp tie. Document what `flatten the overall nested framework of MoNet into a conventional joint optimization one' means at the code level, and if the effect only appears late in training, say so.
• Fix determinism and the VL entry-point crash. Call setup_seed from every main_nl_*.py, register --random_seed in main_nl_vl.py's argparse, align the shipped --alpha in the launch scripts with the paper's $\lambda = 10$, and expose $\gamma$, $\beta_1$, $\beta_2$ as arguments.
• Fix or disclose the VGGSound50 / CREMA-D val-equals-test protocol. Carve out a validation subset from training or disclose in the paper that model selection uses the test set. The same applies to CREMA-D.
• Fix the dataset bookkeeping. Reconcile CREMA-D $7{,}442$ vs. $8{,}185$, correct the MVSA val/test swap between Table 4 and prose, add a citation and split pointer for AVSBench, and fix the README's reference to the non-existent cramed_nl_vl.sh and to the 3D MNIST Dropbox link that ships preprocessed features rather than the raw .h5 the loader needs.

Suggested

• Correct the $w/o$ AMCF interpretation. Table 3 shows Video drops on this ablation; the text should split its analysis by modality rather than say `unimodal performance may slightly improve'.
• Repair Algorithm 1. Insert an explicit forward pass before step 6 so the encoder update clearly uses the current iteration's loss rather than a stale one.
• Fully specify the relative-influence metric and $\delta_a$/$\delta_v$. State the norm convention and any normalization, and describe how the unimodal signals are computed (presumably via the DMSM modality mask $\mathbf{m}^j$).
• Add a per-dataset hyperparameter table. Learning rate, batch size, epoch count, and any dataset-specific choices should appear in the appendix; the current implementation-details paragraph is CREMA-D-centric.
• Ship analysis and plotting scripts. A small analysis/ folder that implements $s^j$, gradient-magnitude logging, stage-tagged per-layer signal logging, and plotting for Figures 3, 5, 6, 7, 8, 9 would make the diagnostic story reproducible with reasonable effort, and would let readers check whether the reconstructed Figure 1(a) balance (which did not favour MoNet under a normalized-accuracy proxy) reflects the paper's Eq. (9) measure or an artifact of the substitution.
• Reconcile the released README with the paper. The current README describes IBML / BIO optimization / TCM task-complexity modulation and reports CREMA-D at $83.33\%$; the paper reports $83.11\%$ and names the method MoNet. A quick pass to align vocabulary and the checkpoint number would remove doubt about artifact identity.
• Discuss the strong/weak identification when it flips. OGM-GE on CREMA-D and MoNet itself both end up with video stronger than audio; a short note on when the `audio = strong' framing applies and how the mechanism behaves once the ranking flips would remove ambiguity.