Detecting the Semantic Fixed Point: A Geometric Framework for Efficient Inference

Summary

This paper proposes Geometric Convergence Early Exit (GCEE), a training-free, vocabulary-independent stopping criterion for autoregressive LLM inference. The conceptual move — recasting Transformer depth as a non-stationary fixed-point iteration and asking "has the last-token hidden state stopped moving?" rather than "is the model confident?" — is a genuinely clarifying reframing of a well-worn problem. Rather than projecting through the LM head at every candidate exit layer (an $O(dV)$ operation that dominates the inference budget for modern LLMs), GCEE monitors two $O(d)$ signals per layer: a normalized step size $\tilde s^{(l)} = \|\mathbf{u}^{(l)}\|/\|\mathbf{h}^{(l)}\|$ and the cosine similarity $c^{(l)}$ between consecutive updates, exiting when both cross fixed thresholds for $K$ consecutive layers past a warmup depth. The empirical hook is a sharp "two-phase" trajectory (large volatile updates in a construction phase; small collinear updates in a propagation phase) that is documented across four benchmarks and eight architectures. On LLaMA-2-7B/13B, GCEE saves ~34% of FLOPs at ~97–99% accuracy retention, and — because the criterion sidesteps the vocabulary-coupled projection — the FLOPs saving actually converts to a 1.45× end-to-end wall-clock speedup, versus 1.14× for entropy-based methods. Appendix B contributes a self-contained theoretical scaffold (LayerNorm Jacobian, $\ell_\infty$ logit bounds, argmax-invariance certificate) that ties the geometric predicate to distributional stability of the next-token softmax. The main conceptual limitation is that GCEE's central quantitative claims rest on definitional choices that the manuscript does not fully expose: the curvature-based saturation layer that grounds Table 5, the correctness split in Table 18, and the error-attribution categories in Table 10 all depend on an unspecified smoothing kernel; the "geometric convergence vs. confidence" comparison in Table 6 uses baseline hyperparameters that are never disclosed; and Appendix B's error bounds are "guiding" only in shape — the numerical thresholds are grid-searched, not derived. The phenomenon is compelling and the method is elegant, but the case for reproducibility of the specific numbers the paper leans on for the framing is weaker than the case for the phenomenon itself.

Strengths

• Conceptual reframing. Moving the exit signal from output-space (confidence, entropy, top-1) to state-space (step size, directional stability) removes the $O(dV)$ bottleneck by construction, and the paper is right to emphasize that FLOPs savings only become speedups when the decision is also cheap. This is a real, non-trivial insight for LLM inference at V > 32k.
• Two-phase phenomenon. The empirical documentation of a sharp construction-to-propagation transition (Figure 1(a,b), Table 14 for effective rank, Table 15 for intrinsic dimension, Table 17 for Attn/FFN takeover) is convincing on its own terms and is broader in scope than the method needs.
• Simple, drop-in criterion. GCEE is one page of code, has no learned components, and works with existing KV-cache infrastructure. Section 3.4's prompt-level exit + truncated-decoding pattern is a sensible engineering choice.
• End-to-end speedup argument. Table 10 makes the FLOPs → wall-clock conversion explicit and separates decision cost from block cost. The 96% efficiency ratio for GCEE vs 77.6% for entropy is the paper's most convincing pragmatic number, and the underlying decomposition (2.5 ms vs <0.1 ms per-layer overhead) directly explains it.
• Theoretical scaffolding. The Appendix B chain — cone-coherent contraction → tail-residual bound → LayerNorm Jacobian → $\ell_\infty$ logit bound → argmax-invariance and MCQ ranking certificates — is more thorough than typical for a method paper, and gives readers the constants (LayerNorm gain, $\sigma_{\min}$, LM-head operator norm) they need to reason about the sufficient conditions.
• Broad diagnostic sweep. Appendix D provides multiple, mostly orthogonal geometric views (SVD spectrum, intrinsic dimension, phase-space trajectory, Attn/FFN decomposition, correctness signatures). Cross-task (Section E.1) and cross-architecture (Section E.2) validations further stress-test the phenomenon.

Weaknesses

• Undisclosed baseline hyperparameters. The main tables (2, 3, 6, 10) compare GCEE against Entropy, Top-1 Confidence, Hidden Similarity, and Unified Skip, but the paper never states the entropy threshold, top-1 probability threshold, hidden-similarity cosine threshold, or the layer selection pattern for Unified Skip. Given that all baselines are being tuned to reach ~21–22 average layers to match GCEE, the accuracy gaps in Tables 2 and 3 are entirely determined by these undisclosed choices.
• Curvature-based $l_{sat}$ underspecified. The reference against which "$|l_{exit} - l_{sat}| = 1.4$" (Table 5), the correctness split (Table 18), the error-attribution categories (Table 10), and the difficulty correlation (Table 1, Figure 1e) are all measured is defined only as "the layer with the maximum slope change in the smoothed log $\tilde s^{(l)}$ trajectory". The smoothing kernel, differentiation scheme, and tie-breaking are unspecified — precisely the choices that can shift $l_{sat}$ by 1–2 layers per sample.
• Threshold-vs-curvature agreement risks circularity. The threshold proxy uses (0.02, 0.85), the same constants used descriptively to characterize the propagation phase in the same paragraph. The r = 0.98 agreement therefore does not fully rule out a "threshold artifact" and is closer to internal consistency than to independent validation.
• Overclaimed theoretical guidance. The introduction claims Appendix B "yields explicit error bounds that guide principled threshold selection". The actual thresholds ($\tau_s = 0.01$, $\tau_c = 0.9$) come from grid search on 1,000 TriviaQA validation samples (Table 13). The bounds are conditional on unmeasured quantities ($\sigma_{\min}$, LM-head operator norm) and do not deliver numerical thresholds; they justify the shape of the criterion.
• Truncated decoding freezes $l_{exit}$ for the whole answer. The prefill-derived exit is reused for every one of up to 50 generated tokens, without a check that per-step $l_{sat}$ remains close to the prefill $l_{sat}$. For QA answers that resolve multi-token entities this is a non-trivial assumption.
• Error-attribution categorization is ambiguous. Premature (l_exit < l_sat − 2), Model-limitation (l_exit ≥ 28), Boundary (otherwise) can both fire for the same sample (e.g., l_exit = 30, l_sat = 20). The manuscript does not state the first-match ordering, and the "Full Model Correct? Yes (recoverable)" label for premature exits is not shown to hold per sample.
• "Rarely induces errors that full-depth inference would have avoided" is measured against wrong denominators. The 1.8% premature-exit rate is per-sample; the 18% is per-error. Combining them into a "rarely" statement obscures the actual excess-error rate, which is on the order of 10% of GCEE errors for a model whose overall EM is ~63.5.
• Fitted contraction ranges partly overlap the construction phase. The reported $\rho \approx 0.86$ ($R^2 > 0.94$) fits are computed over layers 18–32 (7B) and 20–40 (13B), which is broader than the propagation regime for many samples (Figure 1(c) shows 90% saturation within [14, 24]).
• Self-calibrating variant anchored on volatile early layers. The relative threshold $\tilde s \le 0.15 \cdot s_{\text{ref}}$ uses $s_{\text{ref}} = \mathrm{median}(\tilde s^{(1)}, \ldots, \tilde s^{(l_{\min})})$, i.e., the median of the first four layers, which the paper elsewhere calls "large and volatile". No per-task retention is reported, only "within 0.5% of grid-searched performance".
• Cross-architecture universality mixes tasks. BERT-Large and T5-Large are evaluated on SST-2 while the LLaMA/Mistral/Qwen/Phi rows use TriviaQA (Table 20). The universal-range claim $l_{sat}/L \in [0.54, 0.63]$ is therefore not measured on a single task.
• Non-standard diagnostics with unspecified estimators. Table 16's Lyapunov exponent and attractor dimension are estimator-sensitive quantities reported without saying which estimator or hyperparameters were used; the construction-phase attractor dimension is silently blank.
• Small theoretical rough edges. Proposition B.2's scale-invariance ignores $\varepsilon_s$; the LayerNorm eigenvalue $\varepsilon_{\text{ln}}/\sigma^3$ vs the reported spectral norm $1/\sigma$ implicitly requires $\sigma^2 \ge \varepsilon_{\text{ln}}$; Corollary B.11's parallel-projection lower bound can go negative once $k\theta > \pi/2$; the Corollary B.24 factor of 4 in the ranking gap deserves a one-line accounting. None of these break the framework, but they should be tightened.
• Motivational claims presented as facts. "Confidently wrong / hesitantly correct" is asserted without citation; "hallucinations exhibit divergent trajectories" is cited from a single recent preprint; the "Cliff-Plateau-Climb hypothesis" is imported from a vision-Transformer preprint without motivating the carry-over to autoregressive LLMs.

Reproducibility & code

The submission does not link to a public code repository, and the provided codebase/ directory is empty; the review is based on the algorithmic listings (Listings 1, 2) and the appendix. Under those conditions, veritas evidence indicates the method itself is largely reproducible from the listings, but several load-bearing measurement pipelines around it are not.

• Method is reproducible from Listing 1. GCEE's core predicate (Eqs. 4–7) plus Listing 1's code is precise enough that an implementer can reproduce the GCEE row of Tables 2 and 3 to within FP16 kernel noise. Support for the phenomenon-level claims (two-phase structure, effective-rank collapse, ID compression, r = 0.98 between the two saturation detectors) is high once the analysis subsets are fixed.
• Baseline hyperparameters missing. Entropy, Top-1 Confidence, Hidden Similarity, and Unified Skip are compared without disclosed thresholds or skip-layer choices. The full accuracy-vs-GCEE gap in Tables 2, 3, 6, 10 is therefore not directly reproducible — a reviewer cannot check whether the ~5-point margin over entropy is a fair calibration or an artifact of baseline tuning.
• Curvature-based $l_{sat}$ estimator unspecified. The smoothing kernel and slope-change definition that define $l_{sat}$ are not given. Table 5's discrepancy statistics, the correctness split in Table 18, the error-attribution categories in Table 10, and the difficulty correlation panels in Figure 1 all inherit this ambiguity.
• Analysis subsets not seeded. The 2,000-sample Section-2 subset, the 5,000-sample Appendix-D subset, the 1,000-sample TriviaQA validation split, and the 120-sample Figure-1(a,b) subset are all referenced by size but not by seed or index list, and no code is released.
• FLOPs-accounting formula not stated. The 34% / 30–35% FLOPs claim requires a stated model: does it count attention (quadratic in sequence length), KV-cache reads/writes, embedding, and the single LM-head projection at exit? The paper reports the ratio but not the formula.
• Latency protocol under-specified for the headline speedup. The 1.45× vs 1.14× E2E speedup is measured "batch size 1, A100, FP16" without CUDA/PyTorch versions, warmup/iteration counts, sequence-length distribution, or whether the entropy baseline uses a fused LM-head kernel.
• Non-standard diagnostics. Lyapunov exponent, attractor dimension, curvature integral (Table 18), and per-head "active head" threshold (Table 17) rely on estimators/denominators that are not defined; as reported, they are decorative rather than reproducible.
• Truncated-decoding listing is schematic. Listing 2's TruncatedModel omits attention-mask and rotary-position plumbing needed for correct incremental decoding under KV-cache; the paper cites this as its reference implementation, which invites re-implementation errors.
• Self-calibrating variant lacks per-task numbers. "Within 0.5% of grid-searched performance across all tasks" is the only reported number for the variant that is meant to enable near-zero-tuning deployment.

Recommended Changes

Essential

• Release code, seeds, and dataset splits. Publish the GCEE implementation, the 2,000- / 5,000- / 1,000-sample subset seeds, the 120-sample Figure-1(a,b) subset, and the exact FLOPs-accounting formula. Without a code release the manuscript can be reproduced only in outline. (Addresses Weaknesses: undisclosed baseline hyperparameters, curvature-based $l_{sat}$, Section 4 evaluation subsets, and the Reproducibility & code section as a whole.)
• Report the disclosed hyperparameter choices for every baseline in Tables 2, 3, 6, and 10. State the entropy threshold, top-1 probability threshold, hidden-similarity cosine threshold, and Unified Skip layer selection pattern per task, and clarify whether each baseline was per-task-tuned or transferred from prior work. (Addresses: undisclosed baseline hyperparameters.)
• Give an operational definition of the curvature-based $l_{sat}$. State the smoothing kernel and bandwidth, the differentiation scheme, and the tie-breaking rule; report a sensitivity analysis showing that Table 5, Table 10, Table 18, and Figure 1(c,d,e) survive reasonable perturbations of these choices. (Addresses: curvature-based $l_{sat}$ underspecified, error-attribution ambiguity, correctness split.)
• Break the potential circularity in the "r = 0.98" agreement. Either replace the (0.02, 0.85) threshold-proxy with a criterion that does not share constants with the descriptive characterization of the propagation phase, or reframe the r = 0.98 result as an internal-consistency check rather than an independent validation. (Addresses: threshold-vs-curvature agreement risks circularity.)
• Rephrase the theory contribution. State that Appendix B justifies the shape of the criterion and the argmax-invariance certificate, but that the numerical thresholds ($\tau_s$, $\tau_c$, $l_{\min}$, K) are chosen by grid search on 1,000 TriviaQA validation samples. (Addresses: overclaimed theoretical guidance.)
• Add a per-token check for truncated decoding. Report per-generated-token $l_{sat}$ during 30–50-token QA answers, and either (i) show that later tokens remain within retention at the frozen prefill $l_{exit}$ or (ii) add a per-step-exit variant. (Addresses: truncated decoding freezes $l_{exit}$.)

Suggested

• Rework the error-attribution table. State the first-match ordering of Premature / Model limitation / Boundary; verify the "Full Model Correct? Yes (recoverable)" label per sample; and separately report the per-sample premature-error rate rather than the within-error share. (Addresses: error-attribution categorization; "rarely induces errors" overreads.)
• Fit $\rho$ per sample past that sample's $l_{sat}$. Move the geometric-decay fit (Table 10 middle section) out of the aggregate 18–32 / 20–40 windows and into per-sample propagation-only ranges. (Addresses: fitted contraction ranges overlap construction.)
• Report per-task retention for the self-calibrating variant. Add a small table alongside Table 13 and analyze sensitivity to the 0.15 constant and to $l_{\min}$. (Addresses: self-calibrating variant anchored on volatile layers.)
• Report cross-architecture $l_{sat}$ on a common autoregressive task where possible. Add a masked-language-modeling or LM-perplexity view for BERT/T5 so the universal $l_{sat}/L$ range is not measured across mixed tasks. (Addresses: cross-architecture universality mixes tasks.)
• Name the estimators for Lyapunov exponent, attractor dimension, curvature integral, and "active heads > 4%". Either specify the estimator + hyperparameters (Rosenstein / Wolf, Grassberger–Procaccia, ...) or drop the numeric values in favor of qualitative language. (Addresses: non-standard diagnostics with unspecified estimators.)
• Tighten the appendix. Restate Proposition B.2 as approximate invariance with an explicit $\varepsilon_s$ error term; state the $\sigma^2 \ge \varepsilon_{\text{ln}}$ condition under which the LayerNorm spectral norm equals $1/\sigma$; state the regime in which Corollary B.11's parallel projection stays positive; add the one-line accounting for the factor of 4 in Corollary B.24. (Addresses: small theoretical rough edges.)
• Soften motivational claims. Remove or hedge the "confidently wrong / hesitantly correct" dissociation (add a citation or move the evidence forward to Table 6); qualify the hallucination-geometry framing; and either motivate the Cliff-Plateau-Climb carry-over from vision to autoregressive LLMs or drop the load-bearing dependence on that framing. (Addresses: motivational claims presented as facts.)
• Add per-model / per-hyperparameter variance. For Table 13's grid search (N = 1,000), report at least a Wilson interval or three-seed variance so that the choice of default ($\tau_s = 0.010$ vs 0.012) is not driven by sub-1% differences on a single split. (Addresses: reproducibility notes on hyperparameter search.)
• Add software-stack and timing details. State CUDA / PyTorch versions, warmup and measurement iteration counts, and whether the entropy baseline's LM-head projection uses a fused kernel — the 25× decision-latency gap is not intrinsic to $O(dV)$ vs $O(d)$ and depends on all of these. (Addresses: latency protocol under-specified.)