From Text to Forecasts: Bridging Modality Gap with Temporal Evolution Semantic Space

Summary

The paper attacks a real and increasingly visible problem in multimodal time-series forecasting: text embeddings and numerical observations sit in profoundly different representational regimes, and existing "fuse embeddings + attention" recipes leave most of the potentially useful signal on the table. The conceptual move is to insert an intermediate Temporal Evolution Semantic Space between text and numerics — a small, hand-designed vocabulary of interpretable primitives (mean shift, volatility shift, shape, lag/decay) whose ground-truth values are computable from the forecast window itself. A frozen LLM is prompted to classify each primitive from the input text, a confidence gate suppresses low-margin extractions, and the gated primitive embeddings are prepended as prefix tokens to a PatchTST backbone. Because the primitives are numerically verifiable, the LLM classifier can be supervised against ground-truth labels derived from $\psi_k(Y_t)$, which is a genuinely useful idea and makes the framework more principled than most current "prompt-and-fuse" pipelines. Two supporting arguments — a semi-synthetic diagnostic showing that attention over raw text mis-allocates mass and cannot decode qualitative language into numerical gains, and an information-theoretic Rademacher/mutual-information analysis of the semantic bottleneck — situate the design nicely within the literature. Empirically the paper reports strong headline gains on Bitcoin and FNSPID (up to 29% MSE) and modest wins on Electricity, with a runner-up on Environment.

Two concerns pervade the paper. First, the diagnostic that motivates the whole approach is built on a synthetic corpus where the text is generated from the ground-truth future statistics — a maximal-signal setting quite unlike real news — and the primitive-extractor is never evaluated for accuracy on the four real-world corpora. Second, the reported gains lean on point estimates without seed variance, and an independent execution attempt on the released artifact reproduced only the FNSPID column (the Bitcoin / Electricity / Environment data are not shipped and no fetch script is provided); on that one column TESS's advantage collapses to roughly $+3\%$ MSE versus a paper-reported $+20\%$, the Table 2 ablation ordering is reversed, and the confidence gate hurts rather than helps. The reduced-scale reconstruction is a plausible reason magnitudes shrank, so this alone does not falsify the paper — but taken together with the missing seeds, the underspecified "Direct Text" baseline, and the absent LLM-scale ablation, it is currently very hard to attribute the headline improvement to the semantic-bottleneck mechanism rather than to a favourable configuration on one dataset.

Strengths

• Conceptual contribution. Positioning an interpretable, low-cardinality semantic space between text and time series is a clean formulation of the modality-gap problem and separates two failure modes (attention distraction; representational mismatch) that prior fusion work usually confounds.
• Numerical verifiability as a training signal. Because $\psi_k(Y_t)$ is computable from the target window, the primitive classifier and the gating layer can be supervised rather than trained end-to-end from the forecast loss alone. This is a useful design point that gives the LLM extractor a well-posed learning signal.
• Semi-synthetic diagnostic. Even with the caveats below, the FULL / SIGNAL-ONLY / NUMERICAL comparison in Section 3.2 is an unusually clean way to isolate "does redundancy hurt" from "can we decode qualitative language at all", and the framing shapes the reader's expectations for the rest of the paper.
• Theoretical framing. Theorems A.2 and A.5 make the semantic-encoder restriction concrete: $I(Z;Y\mid S)$ is preserved while $I(Z;X\mid S)$ cannot increase, giving a not-increasing information-theoretic generalization bound. Theorem A.8 with its $\sqrt{M/n}$ complexity term (with $M = \sum_k M_k$) is a reasonable formalisation of why a small discrete vocabulary would generalise more easily than raw-text conditioning.
• Non-stationarity subset analysis. Slicing the test set by shape / volatility / distribution-shift subsets is a good sanity check that the reported gains come from the mechanism the paper claims to target.
• Convergence-stability qualitatively confirmed. One qualitative aspect of the paper did survive the independent execution attempt: direct text fusion produced markedly less stable optimisation dynamics than TESS, consistent with the paper's motivation that fusing raw text into a numerical predictor is optimisation-hostile.

Weaknesses

• Diagnostic corpus is a leaked-signal setting. In Section 3.2 the text is generated by an LLM from statistical features of the future window, so any fusion model that could decode text perfectly would essentially see the answer. This is the strongest possible information budget for text and is not representative of real news. The strong conclusion that qualitative language "cannot be decoded into predictive gains" therefore describes fusion failures under best-case-for-text conditions.
• Focus-ratio normalisation unstated. The $R_t < 0$ conclusion depends on whether $R_t$ averages per-token or sums attention mass. Because $|T_\text{sig}| \ll |T_\text{red}|$ by construction, a uniform-attention baseline would already yield $R_t < 0$ under a sum-based ratio. Without the normalisation being spelled out and a random-attention control reported, the "systematic over-attention" interpretation is not well-founded.
• Primitive taxonomy inconsistently enumerated. Abstract lists four coordinate primitives; Section 4.1 states "three categories, four primitives" and folds volatility under distribution shift with a shared table; Figure 8(a) uses the four-primitive labelling. In the shared table, the semantic column is written in mean-level language ("Level unchanged", "Significant increase") but is applied to both $\Delta\mu$ and $r_\sigma$, so STRONG-RISE on $r_\sigma$ reads as a rise in level rather than a jump in variance.
• Shape primitive lacks a well-defined classification rule. The uniqueness claim (v_{t,k} is uniquely determined via $\psi_k$) holds for the threshold-based primitives but not for the shape primitive, whose PEAK / TROUGH / OSCILLATE categories are described informally. Sequences with two sign changes are legal members of both PEAK and OSCILLATE under the given text, so $\psi_\text{shape}$ is not actually a function without an explicit tie-breaking rule.
• Ablation percentages don't reconcile with Table 2 values. Bitcoin (46.2%) and Electricity (22.8%) match (w/o TESS − TESS)/(w/o TESS), i.e. the reduction ratio even though the text says "MSE increasing by …". More seriously, the FNSPID number (29.4%) does not reproduce from any column of Table 2 ($(0.0018-0.0012)/0.0018 = 33.3\%$), and the gating figure 2.6% on FNSPID is even more discrepant with the reported 0.0015 → 0.0012 gap.
• RMSE gain on FNSPID uses the wrong baseline. Pyraformer (0.0383) is the best FNSPID RMSE in Table 1, not TimesNet (0.0385); the 9.9% gain is $(0.0385-0.0347)/0.0385$ and the "strongest baseline TimesNet" label is inaccurate for RMSE.
• Environment bolding contradicts the table caption. Bold red is defined as the best result, but Ours is bolded on Environment despite Nonstationary Transformer being numerically better. The prose concedes runner-up status; the table does not.
• No variance / seed reporting. Several conclusions (Electricity's 0.4% MAE gain; Environment's <1% gap; Bitcoin's Table 2 gains on ~100 test windows) sit within any plausible seed noise. The paper reports point estimates only and does not disclose seed count or aggregation function, even though the Limitations section itself notes possible "fluctuations".
• Bitcoin evaluation set is very small. With 858 total daily observations and input=prediction=7, the test split is on the order of ~100 windows. The most-cited headline gain (29.1% MSE on Bitcoin) rests on this handful of samples with no bootstrap or seed variance reported.
• FNSPID timestep count inconsistent with the stated granularity. 49,165 timesteps over 1999–2023 at daily granularity is ~5–6× the number of days; either the granularity is not truly daily or the row is mis-tabulated.
• Complexity comparison assumes a strawman baseline. The Remark contrasts $\sqrt{M/n}$ against $\sqrt{T/n}$ under $|A_T| = 2^T$, but the actual text-fusion baselines you compare against are not finite-class predictors over $2^T$ binary token configurations; they are continuous-embedding attention models. The favourable scaling comes from characterising the competitor by a bound that does not describe the empirical baselines.
• "Can only tighten" overstates Theorem A.5. From $I_{\tilde p}(Z;X\mid S) \le I_p(Z;X\mid S)$ and equality of $I(Z;Y\mid S)$, the generalization bound is not-increasing, not strictly tightened.
• LLM choice is unablated and central to gain attribution. Everything downstream of the semantic space is conditioned on Qwen3-8B labels supervised against $\psi_k(Y_t)$. A larger or differently-prompted LLM could reproduce the reported behavior — or beat it — and the paper reports neither an LLM sweep nor primitive-extraction accuracy on real-world text, so it is difficult to attribute gains between "the primitives are useful" and "Qwen3-8B is a good future-statistic guesser".
• "Direct text fusion" is undefined. Figures 6 and 7 compare against a Direct Text baseline whose architecture, tokenizer, and fusion mechanism are never specified in Section 5.1. This matters more given that a natural analogue (PatchTST + pooled Qwen3-8B news token) actually descends to a lower loss than TESS at reduced scale.
• Missing oracle-primitive control. An oracle experiment feeding $\psi_k(Y_t)$ (and a noised version) directly to PatchTST would separate the value of the primitives themselves from the value of the LLM extractor; a random-attention control would ground the $R_t < 0$ interpretation. Neither is reported.

Reproducibility & code

An independent run of the released repository was executed, not merely inspected. The core takeaway is that the artifact does not currently support the paper's headline numbers end-to-end: three of the four datasets could not be run at all, and on the one dataset that could, the qualitative story only partially survived.

• Headline aggregates. The up to 29% figure could not be checked because Bitcoin is not shipped; on the one dataset that ran (FNSPID) TESS's advantage over the strongest paper baseline (TimesNet) shrank from +3.3/+20.0/+9.9% (MAE/MSE/RMSE) to +0.4/+3.3/+1.7%, and a no-text unimodal ablation (MSE 0.001334) actually edged out TESS (0.001340). Reduced sample count (a public ~1.5k FNSPID reconstruction vs. the paper's ~40k proprietary split) is a plausible reason magnitudes shrank; direction is preserved, magnitude is not.
• Ablation ordering. The FNSPID w/o TESS / w/o Gating / TESS ordering inverted in the re-run: full TESS was the worst configuration on all three FNSPID metrics (MSE 0.00151 vs. 0.00134 for w/o Gating and 0.00135 for w/o TESS), so the gating gate specifically hurt (-11% MSE) rather than helping by the paper-reported +2.6%. This is confounded by the ver_ours reconstruction (see below), so it is not a clean falsification, but the artifact as published does not reproduce Table 2 on FNSPID.
• Figure 6 convergence. The reproduced FNSPID loss curves show direct fusion descending to a lower training and validation loss than TESS, and TESS if anything more oscillatory in the validation tail — the opposite of the paper's "faster and smoother" narrative. Electricity curves could not be reproduced (no data).
• Semi-synthetic diagnostic (Figures 2–3). The shipped ver_syn_* splits have length-4 ground-truth arrays across all of validation while the harness expects a horizon of 5, which crashes the collate; after heuristic padding to horizon 5 the task becomes trivially solvable (all three variants converge to MSE $\sim 5 \times 10^{-7}$) and the NUMERICAL < SIGNAL-ONLY < FULL ordering cannot be recovered. Fixing the shipped horizons — or releasing the exact tokenised splits — is a prerequisite for anyone to check the motivating diagnostic.
• Focus-ratio diagnostic. Regenerating Figure 2 requires a token-level fusion variant; the shipped MultiModal_Baseline pools news to a single token, so per-token cross-attention weights are uniformly 1.0 and $R_t$ is undefined. The specific model class that produced Figure 2 is absent from the release.
• Core TESS pipeline. The ver_ours fusion input and the text_quality gating label are not built by the release and had to be reimplemented from the paper text; the shape-extraction script silently ignores batch_size and never completes at scale; and the quality-aware model class is unreachable through the registry due to a name-normalisation vs. filename mismatch. All of Tables 1–2 and Figures 5–8 therefore currently require reconstructing the central data pipeline before a single TESS number can be produced.
• Datasets and Qwen3-8B embeddings. None of Bitcoin / FNSPID / Electricity / Environment ships with the code, and there is no fetch script; the Qwen3-8B embedding files the framework consumes (ver_gen1, ver_camf/embedding_qwen/all_embeddings.pt) are also absent. A public FNSPID reconstruction was the only path to any measurement — precisely why Bitcoin / Electricity / Environment stayed unattempted.
• Baseline mapping and protocol. The repo has a MultiModal_Baseline family and a timellm.py, but the paper labels are TimeLLM, ChatTime, NewsForecasting; there is no README or config note tying rows to (model_file, config_yaml) pairs. Seed count, seed values, and aggregation function behind each cell of Tables 1 and 2 are stated neither in Section 5.1 nor in the code.

Recommended Changes

Essential

• Release a working TESS pipeline. Populate the missing TESS model class, expose the quality-aware variant through the model registry (the current filename / name-normalisation mismatch makes it unreachable), fix the batched shape-extraction script so it honours batch_size, and add CLI flags or dedicated YAMLs for w/o TESS, w/o Gating, and per-primitive removals so Tables 1–2 and Figure 8(a) can be regenerated by a third party.
• Ship the datasets or a preprocessing pipeline. Provide either processed tensors for Bitcoin / FNSPID / Electricity / Environment (with license attribution) or a scripted pipeline that starts from a downloadable source and produces the trainer inputs, including the Qwen3-8B embedding files. This is what blocked reproducing the Bitcoin (headline 29%), Electricity, and Environment numbers.
• Fix the shipped semi-synthetic splits. The ver_syn_* labels have a length-4 horizon while the harness expects 5, which crashes the collate; after ad-hoc padding the task becomes trivially solvable and the FULL / SIGNAL-ONLY / NUMERICAL ordering vanishes. Re-release the exact splits used for Figures 2–3, or the construction pipeline.
• Report seed variance and averaging protocol. Add mean±std across at least three seeds for every cell of Tables 1–2 and the non-stationary subset analysis, and disclose the seed values in Section 5.1. This is especially important for Environment (<1% gap), Electricity (0.4% MAE), and Bitcoin (small evaluation set) — and for defending FNSPID's 20% MSE gap given that an independent run reproduced only ~3%.
• Replace GPT-5.2 with an actually available model. Pin the diagnostic corpus generator to a dated/versioned model (or an open-weight LLM) and release the prompts and the generated texts so Section 3.2's attention-distraction and representational-mismatch analyses can be checked.
• Report primitive-extraction accuracy on real data. Add a per-primitive accuracy table (Qwen3-8B classification vs. $\psi_k(Y_t)$) on each of the four real datasets, and at least one LLM scale / family ablation. This is what the gating soft-weight boxplot in Figure 8(b) is implicitly relying on and what the reproducibility gap on gating (opposite sign on FNSPID) most needs to disambiguate.
• Formalise $\psi_\text{shape}$ and re-audit uniqueness. Give an explicit algorithm (decision list with strict priority) that partitions sign sequences into ASCEND / DESCEND / PEAK / TROUGH / OSCILLATE, so the shape primitive is well-defined and the "uniquely determined by $\psi_k$" claim actually holds.
• Recompute the FNSPID ablation percentages. Either recompute the 29.4% and 2.6% figures from the numbers actually printed in Table 2 (which yield 33.3% and ~20–25%), or publish the pre-rounding raw MSE values, and pick either "MSE increases by X% on removal" or "X% MSE reduction from adding TESS" and use that convention consistently.

Suggested

• Add an oracle-primitive baseline. Feed $\psi_k(Y_t)$ (and a noisy version) directly to PatchTST to isolate how much of TESS's gain comes from the primitives themselves vs. the LLM extractor.
• Clarify the focus ratio. Add the formal definition of $R_t$ in the main text, state whether it is per-token or mass-based, and include a random-attention control so $R_t < 0$ is interpretable. Also ship the token-level fusion variant the diagnostic requires — the released MultiModal_Baseline pools news to a single token and produces trivial attention.
• Reconcile the primitive taxonomy across abstract, Section 4.1, and Figure 8(a). Pick one grouping and use it end-to-end, and split the STABLE/RISE/DROP semantics table so $r_\sigma$ and $\Delta\mu$ are labelled in their own variable's language.
• Fix Table 1 conventions. Bold only the actual best result per the caption (unbold Ours on Environment), and label the per-metric strongest baseline correctly (Pyraformer for FNSPID RMSE, not TimesNet).
• Explain the FNSPID timestep count or correct it, so the row is consistent with a daily granularity over 1999–2023.
• Weaken the complexity claim. Reframe the Remark as a comparison against a discrete token-level baseline of stated cardinality, or motivate why the continuous attention baselines you compare against are meaningfully characterised by the $\sqrt{T/n}$ term.
• Weaken "can only tighten" to "cannot loosen", or state the strict-tightening conditions explicitly.
• Specify the "Direct Text" baseline used in Figures 6 and 7 (architecture, text encoder, fusion, hyperparameters), and add a baselines.md mapping every paper-named baseline in Table 1 to its (model_file, config_yaml).
• Qualify the "up to 29%" framing. Add a representative aggregate (mean gain across datasets/metrics, or per-dataset gains) alongside the maximum figure in the abstract and conclusion.
• Case-study transparency. Annotate each panel of Figure 7 with (predicted primitive, ground-truth primitive, gating weight) so the reader can distinguish "primitive was extracted correctly" from "forecast tracked the ground truth".
• Fix the $\bar u$ notation in the shape primitive: the inter-patch trend should use $\bar u_{i+1} - \bar u_i$ (both terms with the bar).