Modeling Hierarchical Thinking in Large Reasoning Models

Summary

The paper proposes a Finite State Machine (FSM) abstraction of Large Reasoning Models' Chain-of-Thought as a sentence-level trajectory over six functional states — initialization, deduction, augmentation, uncertainty, backtracking, closure — and derives a training-free control policy over that abstraction. Sentences are mapped to states via a GPT-4o-mini annotator, outcome-conditioned transition matrices $T^{(correct)}$ and $T^{(incorrect)}$ are estimated, and their difference defines a Transition Advantage Matrix $R$. Treating $R$ as an immediate reward on a small deterministic MDP (action = choose next state), the authors run 100 iterations of Q-value iteration ($\gamma = 0.9$, reward clipping) to obtain long-horizon transition utilities. At inference, at sentence boundaries only, a state encoder plus current/next-state classifiers gate whether to intervene; if intervention is warranted, a contrastive difference-of-means transition vector is injected orthogonally into the residual stream. The conceptual move — reframing activation steering as planning over discrete cognitive-state transitions rather than local per-behavior control — is a genuinely useful lens on the CoT trajectory. Headline claims are +13pp on AIME25 for gpt-oss-20b and roughly $25\times$ fewer interventions than greedy on MATH-500 with GPT-L.

The main conceptual limitation is that $R$ and the resulting policy are correlational: $R$ is a marginal difference between success and failure transition frequencies, so trajectories that spontaneously landed in "positive-advantage" transitions may have done so because they were already on a good path, not the other way around. Appendix E acknowledges this, but the body leans on a causal-reward framing. Compounding this, main-table numbers are single-seed with no reported variance, the emblematic AIME25 result covers 30 problems (so +13pp is ~4 items), several method hyperparameters are pinned only to intervals, and an executed reconstruction of the pipeline from the paper revealed that the released repository does not contain the algorithmic components at all, that the reward-clip specification is ambiguous enough to admit a non-working implementation, and that both the $\kappa=0.89$ annotation-agreement and >90% state-classifier accuracy figures depend implicitly on the specific (gated) annotator model. The FSM lens and Q-value planning framing are worth publishing, but the empirical evidence sits closer to a correlational demonstration than the "planning-aware control" framing suggests.

Strengths

• Conceptual contribution. Recasting activation steering as state-transition planning over a compact FSM is a clean, novel synthesis. It moves the field beyond per-behavior suppression (Chen et al., 2025a; Venhoff et al., 2025) toward a global, multi-step policy view of CoT.
• Interpretable abstraction. The six-state taxonomy is small enough to reason about, connects to classical problem-solving frameworks (Polya, Schoenfeld), and yields per-model advantage matrices (Appendix N) whose structure is directly interpretable and structurally reproducible in a reimplementation (the 6×6 layout, state ordering, diverging colour scale, and mix of signed entries all come out as advertised).
• Sparse-intervention design. Confidence-gated Q-gap thresholding plus stuck-detection is a sensible mechanism for making interventions strategic rather than continuous. In a reconstructed pipeline the direction of the efficiency claim was strongly borne out: Q-Value intervened ~62× less than greedy on a substitute model (DeepSeek-1.5B on a MATH-500 subset), qualitatively matching the paper's ~25× reduction narrative.
• Complementary control panels. The prompt-based baselines (Appendix D), cross-model transfer (Table 2), second-order Markov analysis (Appendix C), diversity/fluency check (Appendix F), confusion-matrix intervention analysis (Appendix H), and boundary-detection diagnostics (Appendix I) each address a natural methodological concern.
• Honest limitations section. Appendix E is candid about the correlational nature of $R$, taxonomy sensitivity, and the intervention mechanism's fragility — well beyond what many activation-steering papers offer.

Weaknesses

• Correlational $R$ used as if causal. $R$ records which transitions are over-represented in successful trajectories, but Sections 3 and 5 build a Q-value control policy on top of it and describe the resulting gains as though the mechanism were causal. Appendix E notes the caveat; the body should be as candid, and the Rescue/Harm analysis in Appendix H should be foregrounded as the actual causal probe.
• Single-seed reporting, small headline benchmark. Every main-table cell uses seed = 42, and the emblematic AIME25 +13pp gain is a 4-item flip on 30 problems. Sub-3pp gains on GPQA Diamond, MATH-500, and GSM8K are all within plausible single-seed noise; without at least 3–5 seeds and paired-bootstrap intervals, most Table 1 pairwise comparisons cannot support the qualitative claims ("consistent improvements", "$25\times$ fewer interventions") they are used to justify.
• Over-generalized claims contradicted by own tables. "All of our steering methods improve accuracy" is contradicted by greedy regressions in five Table 1 cells; "$Q$-Value steering consistently outperforms prompt-based guidance" is contradicted by Table 5 on GPQA Diamond and GSM8K (QWEN); "The accuracy degradation is modest, while the intervention cost is substantially higher" for cross-model transfer omits that Greedy cross-model actually outperforms the model-specific baseline in Table 2.
• Hyperparameter/notation underspecification. The Q-gap threshold reuses the FSM symbol $\delta$; steering-strength ranges are inconsistent between Sections 3.3 ($\beta \in [0.1, 1.2]$) and 5 ($\alpha \in [0.1, 1.0]$); the reward-clip bound $c \in [0.2, 0.3]$ and the floor $\beta$ are intervals rather than pinned per-experiment values; the weighted-steering candidate-set threshold $\tau = 0$ makes the filter vacuous. These make the pipeline hard to specify exactly even with unlimited compute.
• Definitional inconsistencies. The FSM 5-tuple's transition function $\delta: Q \times \Sigma \to Q$ is written as deterministic but described in prose as mapping to a set of successors, and the rest of the paper treats transitions probabilistically. The positive-set definition for a specific transition $(u \to v)$ is described "across all ordered state pairs with $u \neq v$," which is contradictory. The "orthogonal" injection is only approximately orthogonal under the stated normalization ($\varepsilon$ added to $\lVert h \rVert_2$). Empirically the FSM's closure-terminal and init-start structural assumptions do not hold either: the reproduced advantage matrices show non-zero closure outgoing and init incoming transitions.
• Diagnostic reporting selectively softened. The 84–90% false-boundary rate is not accompanied by a denominator definition; the "suppresses the majority" claim omits that GPT-L's suppression rate is only 35.6%; QWEN's slip-through of 18.9% is described as "low" alongside 0.1%; "rescue rates consistently exceed harm rates" glosses over that absolute rescue on MATH-500 QWEN is 8.9%.
• Speculative mechanism claims. "$Q$-Value steering compresses reasoning by increasing information density" is never defined or measured, and is not uniformly supported by the token-count numbers (e.g., AIME25 GPT-L Q-value 2200 vs 1725 default). The DeepSeek-1.5B result is explained by an unmeasured "knowledge capacity ceiling" for which no headroom estimate is provided.
• Illustrative annotation contains a mislabel. In Appendix M, "Now coordinates: let A at 0." is labeled augment-fact, but this is a coordinate-frame modelling choice, not fact recall (essentially the same span earlier is augment-plan). Because this is the showcased example, it undercuts confidence in the automated labels that drive $R$.
• Markov "empirical validation" is thin. Appendix C's sign-agreement is 82.1% on only 78 triplets, and Table 3/4 GFO deltas are small enough to reflect insufficient data for $R^2$ estimation rather than genuine first-order sufficiency. A held-out log-likelihood comparison of first- vs second-order transition models would be a much stronger check.
• Reliability of state labels is fragile to annotator choice. The paper's downstream pipeline (transition statistics, classifier training data, Q-value estimates) depends entirely on GPT-4o-mini's six-state labels. A from-scratch reconstruction with a competent open-weights substitute (mistral-small) collapsed inter-annotator $\kappa$ to ~0.23 and drove state-classifier accuracy from the claimed >90% down to ~57%. This means the >90% classifier ceiling implicitly assumes GPT-4o-mini-quality annotation — a dependency not disclosed in the reliability discussion.

Reproducibility & code

• Empty released repository. The paper advertises https://github.com/shahariar-shibli/CoT-FSM as the code home, but on inspection at replication time it contained only README.md and .gitattributes — no annotation pipeline, no advantage-matrix construction, no classifier training script, no Q-value iteration or steering-injection code, no sentence-boundary detector, no per-benchmark grader. Every algorithmic component had to be reconstructed from the paper's prose to produce any evidence at all.
• Reward-clip specification admits a non-working implementation. The natural reading of "reward clipping ... $c \in [0.2, 0.3]$" as a positive-only band clip(R, [0.2, 0.3]) flattens the reward and prevents Q-Value from ever steering. Only after switching to the symmetric form clip(R, [-c, +c]) did the headline Q-Value mechanism become active. Because no author code exists to disambiguate, a future reproducer can easily land on a silent null result. The notation $[-c, +c]$ in the equation should be reinforced with unambiguous prose and a reference implementation.
• Hyperparameters given as intervals. Reward clipping $c \in [0.2, 0.3]$, the steering-strength floor $\beta \in [0.1, 1.2]$, and (per Section 5) $\alpha \in [0.1, 1.0]$ are all stated as ranges without a per-experiment selection rule. Since $Q$ is bootstrapped from $R_{\text{clip}}$ and $\alpha$ scales the injection magnitude, the resulting Q-table and steered accuracy are not recoverable from the text.
• No intermediate artifacts released. The $R$ matrices per (model, benchmark), the $Q$-tables, the state encoder / current+next classifier weights, and the annotated CoT trajectories are all absent. Even the definitional Figure 4–5 heatmaps cannot be verified without them, and they would be cheap to release even without the training/inference stack. A reconstructed pipeline confirms the structural form of Figures 4–5 (6×6, diverging colour scale, correct state ordering, signed entries) but numerical values necessarily disagree because they depend on the annotator and corpus.
• Answer graders undocumented. The MATH-500 boxed-LaTeX equivalence, AIME25 integer extraction, GPQA Diamond letter extraction, and GSM8K numeric equality routines are not described. On tight-margin benchmarks (GPQA Diamond, MATH-500) a single grader disagreement is comparable to some of the reported gains.
• gpt-oss-20b reasoning effort is not operationalised. The emblematic AIME25 headline uses gpt-oss-20b (low), but the specific reasoning-effort control (system-string, decoding parameter, or provider API knob) is never pinned. The exact configuration matters for reproducing the +13pp result.
• Single seed for all main-table cells. The paper reports seed = 42; there is no multi-seed variance table. Reproducers cannot judge whether their re-decoded numbers match within noise.
• Prompt-template ambiguity in Cross-Model Guidance. Rule 2(a) of the Cross-Model template is the fragment "guess or suspect a mistake." with no Do or Do NOT prefix. Because this template drives Table 5's Cross-Model column, the ambiguity makes those cells non-reproducible without correction.
• Annotation reliability check is thin and gated. Cohen's $\kappa = 0.89$ is computed on 10% of AIME25 (~3 problems) and 10% of GPQA Diamond (~20 problems) and depends on GPT-4o-mini, a gated proprietary model. The annotated subset, annotator instructions, and per-state agreement are not released, so the kappa cannot be independently validated. In a reconstructed pipeline the LLM-vs-LLM proxy $\kappa$ collapsed to ~0.23, exposing the six-state labeling as annotator-dependent.
• State-classifier >90% claim is not robust to annotator drift. With a substitute annotator (mistral-small; annotation $\kappa \approx 0.23$) and the paper's 80/20 classifier training on ~30k boundaries, both classifiers land at ~57% test accuracy — well below the >90% claim and below the confidence-gate threshold (0.90) that governs whether to steer. The paper should disclose that the >90% number is contingent on annotation quality, not on data volume.
• Cross-model transfer mechanics ambiguous. Table 2 transfers "QWEN's advantage matrix to GPT-L," but it is not stated whether the transferred object is only the 6×6 $R$ (with GPT-L-specific steering vectors) or also the steering vectors themselves — an important distinction because activation directions are not architecturally interchangeable.
• Hardware footprint of the steering pipeline is undisclosed. Activation steering requires HuggingFace-level access to the residual stream; loading gpt-oss-20b (MXFP4) into HF transformers triggers dequantization to bf16 (~40 GB), which does not fit on a single A100 with typical shared-cluster headroom. Since the marquee AIME25/MATH-500 cells are on this model, a one-line VRAM disclosure per model would meaningfully aid reproduction planning.

Recommended Changes

Essential

• Populate the released repository. Ship the annotation pipeline, $R$-matrix construction, state-encoder/classifier training, Q-value iteration, orthogonal-injection code, sentence-boundary detector, and per-benchmark answer graders — the components the "Empty released repository" weakness names — before camera-ready.
• Disambiguate the reward-clip semantics in prose and code. Rewrite the reward-clipping equation with an explicit note that the clip is symmetric about zero (i.e., R_clip = clip(R, -c, +c)), and include a reference implementation, addressing the "Reward-clip specification admits a non-working implementation" reproducibility issue.
• Report per-seed variance for Table 1. Run at least 3–5 seeds per (model, benchmark) cell, report mean $\pm$ std, and add paired-bootstrap confidence intervals so that the "consistent improvements" and "$25\times$ fewer interventions" claims (and the "single-seed reporting" weakness) can be judged statistically. AIME25 in particular should have per-item flip counts published.
• Soften the causal framing of $R$-driven steering. Rewrite Section 3.3 and Section 5 to make explicit that $R$ is a correlational difference of marginals and that intervention gains are net effects (Appendix H), addressing the "correlational $R$ used as if causal" weakness. Move the Rescue/Harm analysis into the main body as the primary causal probe.
• Retract or qualify the overreaching claims. Rewrite "all of our steering methods improve accuracy" (contradicted for greedy in Table 1), "$Q$-Value steering consistently outperforms prompt-based guidance" (contradicted on GPQA/GSM8K QWEN in Table 5), and the cross-model summary in Table 2 (Greedy cross-model beats model-specific) to match the tables. This addresses the "over-generalized claims" weakness.
• Pin the interval hyperparameters. Publish a small table listing the exact $c$, $\beta$, layer $\ell$, $\alpha$, and $\delta$ used per (model, benchmark, method) in Table 1, resolving the "Hyperparameters given as intervals" issue.
• Release intermediate artifacts. Ship the per-(model, benchmark) $R$ matrices, Q-tables, state-encoder/classifier weights, and annotated CoT trajectories, so the definitional objects and Figures 4–5 can be checked without rerunning any model — the "No intermediate artifacts" reproducibility issue.
• Document the answer grader per benchmark. Add a subsection specifying the MATH-500, AIME25, GPQA Diamond, and GSM8K extraction/equivalence procedures, and release the grader script, addressing "Answer graders undocumented."
• Operationalise gpt-oss-20b reasoning effort. One-sentence disclosure of the "low"/"medium" reasoning-effort configuration (system string, provider knob, or decoding setting) used for the emblematic AIME25 result.
• Fix the Cross-Model Guidance prompt template. Restore the missing Do / Do NOT in rule 2(a) so Table 5's Cross-Model column is reproducible, addressing "Prompt-template ambiguity in Cross-Model Guidance."
• Disclose the annotator dependency and evaluate a robustness proxy. State that GPT-4o-mini is a hard requirement (or release an open-weights annotator prompt+model that reproduces the labels), and add a sensitivity check showing how state-classifier accuracy and Q-Value gains degrade under a substitute annotator — the "Reliability of state labels" and "State-classifier >90% claim" issues.

Suggested

• Fix the FSM 5-tuple signature. Change $\delta : Q \times \Sigma \to Q$ to a nondeterministic/probabilistic form (e.g. $\delta : Q \times \Sigma \to 2^Q$ or $\Delta(Q)$) so the formal object matches the probabilistic transition machinery, addressing the "Definitional inconsistencies" weakness.
• Rename the Q-gap threshold. Use $\theta_Q$ or $\tau_Q$ instead of reusing $\delta$, to remove the notation collision noted under "Hyperparameter/notation underspecification."
• Drop or fix $\tau = 0$ in weighted steering. Either replace with a small positive value and report sensitivity, or remove the $\tau$ machinery entirely (currently vacuous).
• Reconcile $\alpha$ vs $\beta$ ranges. Clarify whether the reported $\alpha \in [0.1, 1.0]$ (Section 5) and the adaptive floor $\beta \in [0.1, 1.2]$ (Section 3.3) refer to the same quantity, and if so pick one range.
• Reconcile FSM structural assumptions with empirical $R$. Either zero out the closure row and init column of $R$ (and justify) or acknowledge in the FSM formalization that both states have non-trivial transitions in practice (e.g., self-verification masquerading as closure, per Appendix I.(a)), addressing the empirical mismatch called out under "Definitional inconsistencies."
• Correct or clarify the annotated example. Relabel "Now coordinates: let A at 0." (currently augment-fact) to be consistent with the equivalent augment-plan span earlier in the same trajectory, addressing "Illustrative annotation contains a mislabel."
• Report the effective (post-suppression) false-boundary rate. Add a single number per model for the rate of false triggers that survive the confidence gate, so the diagnostic reads as diagnostic rather than catastrophic, addressing "Diagnostic reporting selectively softened."
• Replace "information density" language with a measurement. Either drop the density explanation of Q-value's token behaviour or define and report a concrete metric (e.g., successful sub-goal density per token), addressing "Speculative mechanism claims."
• Strengthen the Markov validation. Add a held-out log-likelihood comparison of first- vs second-order transition models on the annotated trajectory corpus, replacing sign-agreement alone as the primary empirical check.
• Release annotation guidelines and the annotated subset. Publish the 10% AIME25/GPQA subset, the rater instructions, and per-state agreement so the Cohen's $\kappa = 0.89$ number can be independently validated.
• Clarify cross-model transfer mechanics. State whether transfer includes only $R$ or also the steering vectors, and (ideally) run both variants to disentangle them.
• Investigate the QWEN/GSM8K default anomaly. Diagnose whether the 78.77% default is an extraction/formatting artifact before citing the 78.77 → 94.77 prompt-guidance jump as evidence.
• Disclose steering-pipeline VRAM footprints. Add per-model VRAM footprints for the HF-loaded steering path (in particular, the ~40 GB bf16 footprint of dequantized gpt-oss-20b), addressing the "Hardware footprint" reproducibility issue.
• Note the GPT-4o-mini temperature convention. Replace 1 × 10^{-19} with temperature = 0 (or explain why 0 was avoided), addressing the "GPT-4o-mini temperature" reproducibility note.