The Signal is in the Steps: Local Scoring for Reasoning Data Selection

Summary

This paper studies response selection for reasoning distillation: given a prompt and several answer-correct candidate solutions from different long-form reasoning teachers, which trace should a student model be trained on? The starting point is GRAPE (Zhang et al., 2025), which selects by global average log-probability under the student (GALP). The authors argue that GALP, developed for short, single-teacher instruction data, breaks down in the long-and-mixed-teacher regime for two connected reasons: token-average likelihood is dominated by abundant discourse tokens rather than by sparse reasoning-critical ones, and long global contexts inflate probabilities via self-conditioning. Empirically, GALP inverts the teacher ranking (Qwen3-32B has the highest GALP but yields the worst SFT student) and can underperform random selection when candidates come from a mixed pool. The proposed remedy is Local Average Log Probability (LALP): segment each candidate into reasoning steps and score each step under only a small window of preceding steps, then average across steps. An embedding-based coverage analysis motivates the choice: test-time reasoning steps are densely covered by training data while whole trajectories are not, so scoring at the step level aligns the selection target with the granularity at which students seem to generalise. Across math (LIMO, three reasoning teachers), science (GPQA-Diamond), and code (LiveCodeBench), LALP beats GALP by up to 9.4 points and outperforms every likelihood-based baseline. Token-attribution, position-wise self-conditioning gaps, a loss-vs-generalisation dissociation, and a toy theoretical model give the story a coherent conceptual arc. The main limitations are (i) statistical: single-seed runs with no confidence intervals on 30–200-item benchmarks leave several secondary claims within seed noise; (ii) the coverage argument is confounded by the far larger step-vs-trajectory neighbourhood pool; (iii) key methodological details (embedder, segmenter decoding, context window $k$, SFT hyperparameters) are under-specified despite a code release; and (iv) the baseline set is restricted to likelihood variants — no length-controlled or diversity-based comparator isolates locality from GALP's verbosity bias.

Strengths

• Clean conceptual move. Reframing the selection target from trajectory likelihood to step-level local likelihood is a compact idea, well-connected to the intuition that reasoning generalises by recombining reusable steps rather than by memorising trajectories.
• Empirical scope. LALP is tested on two Qwen students of very different sizes, three reasoning teachers with genuinely different styles, seven math benchmarks, plus GPQA-Diamond (science) and LiveCodeBench (code); the ranking-reversal, mixed-pool, teacher-selection, and cross-domain results are individually consistent with the same story.
• Mechanistic diagnostics. Token-attribution mass (Table 4), the position-wise self-conditioning gap (Figure 5), and the loss/generalisation dissociation (Figure 6) go beyond a black-box accuracy comparison and give the improvement a plausible mechanism.
• Robustness ablations. The context-window sweep (Figure 3, Figure 9, Table 11), the segmenter ablation (Table 21, LALP vs NLTK vs \n\n), the verbosity-only baselines (Most/Least Steps), and the expanded five-teacher pool (Table 22) each rule out plausible alternative explanations for LALP's gain, though the expanded-pool ablation deserves clarification (see Weaknesses).
• Practical relevance. Teacher selection with as few as 200 prompts reliably recovers the full-dataset ranking, and the pipeline is straightforward to bolt onto an existing SFT recipe.

Weaknesses

• Coverage argument is confounded by pool cardinality. The nearest-neighbour coverage comparison uses $\max$ cosine similarity at two levels whose candidate pools differ by roughly two orders of magnitude (each test step is matched against ~N × avg-steps neighbours, each test trajectory against N). Max similarity is monotonically non-decreasing in pool size, so an unknown fraction of the 0.935 vs 0.760 gap simply reflects that. Because this coverage claim is the load-bearing motivation for the entire method, it needs a size-controlled comparison.
• Single-seed statistics on small benchmarks. AIME25 and CN_MATH24 have 30 items each; GPQA-Diamond has 198; a single item flip is 3.3–5 points. No seeds are reported for SFT and no confidence intervals for evaluation. The headline +9.4-point AVG on the 32B student is comfortably outside typical noise, but the GALP-vs-Random inversion (1.9 pts), the LALP-vs-best-single-teacher gap (0.7 pts), and the LCB-hard gap (2.9 pts) are each in a range where seed variance could plausibly flip the sign.
• Fluency-trap theorem does not match the deployed setting. Section 3.1 restricts candidates to answer-correct traces, but Theorem A.3 (invoked in Section 7.4.1 as the mechanism for GALP failure) formalises GALP's failure as "systematically preferring incorrect responses." The mechanism the theory formalises is not the failure mode the experiments exhibit.
• Locality is claimed as causal but shown only via correlation with GALP. Figure 3's $\alpha$-sweep shows that LALP ranking converges to GALP ranking as $\alpha \to 1$. This is consistent with locality being the driver, but also with alternative explanations (self-conditioning saturates with long context, verbosity dominates GALP). A control that isolates locality from length would strengthen the causal story.
• Baseline set is narrow. All comparators are likelihood-based (GALP, Local Highest, Local Lowest, Random). Length-normalised likelihood, per-token entropy, or a shortest-correct-trace heuristic would test whether the gain is specifically about locality rather than about avoiding GALP's verbosity/self-conditioning inflation.
• GRAPE-as-baseline framing. GRAPE was designed for short, single-teacher instruction data. Framing LALP against GALP as "fluency trap" and "systematically misranked" reads as if GALP were a proposed long-reasoning selector; softer framing ("does not extend to this regime") would be more accurate.
• Empirical overclaims.
  • The abstract's "consistently improves … by a large margin" is fair for math and science but oversells the coding gains (Table 17 shows a regression on LCB-easy relative to the base model and modest +2.9 on hard).
  • The GPQA-D comparison to LIMO-32B/Sky-T1/OpenThinker2 (Table 20) is not apples-to-apples: training data, prompt set, epoch count and evaluation harness all differ; attributing the gap to "data curation" needs a controlled comparison.
  • The "LALP > best single teacher" claim (0.726 vs 0.719) is a 0.7-point aggregate difference from single-seed runs; "exploits complementary strengths" is a strong causal reading of a small delta.
• Notation and cross-reference slips. LALP is defined with integer window $k$ but the ablation uses fractional $\alpha$; the mapping is not given. The main text cites 0.719 to "Table 2," which does not contain a per-teacher row (that is Table 9). "9.4%" throughout the paper is percentage points, not relative percent. Teacher "Qwen2.5-72B-Instruct" is listed with a Qwen2.5-32B-Instruct URL. Table 8 heads both blocks with "Qwen2.5-32B-Instruct." Table 19 caption promises two students; only one is shown. Table 20 AVG excludes GPQA without saying so. Two Motivation paragraphs open Appendix A back-to-back. These are individually minor but numerous.
• Table 22 LALP row is bit-identical to Table 21 (0.726/0.440). Either the LALP selection set is genuinely unchanged when two non-reasoning teachers are added to the pool — an interesting robustness fact that deserves to be stated — or the row is a copy-paste. As written the reader cannot tell.

Reproducibility & code

• Embedding model for step/trajectory coverage is unnamed. The 0.935 vs 0.760 coverage figures depend on the choice of embedder, pooling, and normalisation. Section 5 and Appendices B/C.8 describe the procedure but the model itself is absent from the text; the released repository would need to make this explicit before those numbers can be regenerated.
• Token-attribution categories are illustrated, not specified. Table 4's 42.3/31.2/12.8/13.7 → 18.7/48.6/22.4/10.3 shift requires an exact lexicon or classifier, plus rules for BPE sub-tokens, LaTeX macros, and mixed tokens; only example words are given. The corpus size is also stated inconsistently (500 in Section 7.4.1 vs 440 in Figure 4).
• Self-conditioning micro-examples omit the student model and full response context. The "177," "1228," and "come in pairs" log-probs (-1.83, -0.10, -0.00003, etc.) depend on tokenizer digit-splitting and on which student is scoring; the passage says neither.
• Main-table LALP window $k$ is not stated in Section 7.2. Appendix C.7 finds a peak at $k=4$–$5$ for the 32B student (0.731), adjacent to the headline 0.726, but the paper never names the exact $k$ used for Tables 3, 10, 16, 17.
• SFT and evaluation hyperparameters listed as "either/or" without mapping. Tables 5–7 give ranges ("1/2," "5e-6/1e-5," "10/15," temperature "0.0/0.6," pass@"1/8") without saying which experiment used which. Given no reported seeds, this leaves each headline number materially under-determined.
• GLM-4.5-Air segmentation decoding is unspecified. Segmentation is upstream of every LALP number in the main tables, but the paper doesn't give the segmenter's temperature, top-p, or retry policy on malformed JSON.
• Anonymised code release is a plus, but the above gaps together mean that a reader with the released code still can't uniquely reproduce Section 5's coverage analysis, Section 7.4.1's attribution table, the individual self-conditioning examples, or the exact main-table numbers.

Recommended Changes

Essential

• Add per-seed variance to the headline claims. Run at least three SFT seeds for the mixed-teacher and teacher-selection experiments and report a mean ± std on the AVG column. Mark which LALP-vs-GALP and GALP-vs-Random gaps survive that noise. This directly addresses the Single-seed statistics weakness and the GALP-vs-Random interpretive overreach.
• Size-control the coverage analysis. For the step vs trajectory comparison in Section 5, either (i) equate the number of candidate neighbours (e.g., top-1 among a size-matched sample of steps) or (ii) provide a step-shuffled trajectory control. Report how much of the 0.935 vs 0.760 gap remains once pool cardinality is controlled. This addresses Coverage argument is confounded by pool cardinality.
• Reconcile Theorem A.3 with the all-correct setting. Either (i) reframe the theorem in terms of higher- vs. lower-quality reasoning between two correct traces, or (ii) drop the "incorrect responses" framing from Section 7.4.1 and explain what the theorem actually implies for a within-correct selection problem. This addresses Fluency-trap theorem does not match the deployed setting.
• State the LALP window $k$, embedder, segmenter decoding, and SFT hyperparameters used for each table. Add a "Configuration used" caption row (or an appendix table) that maps each experiment to a single row of Tables 5–7 plus the exact $k$, the exact embedder for Figure 2, and the GLM-4.5-Air decoding config. This addresses the Main-table $k$, Hyperparameter listing, Embedding model, and Segmenter decoding reproducibility weaknesses in one shot.
• Add a length-controlled or non-likelihood baseline. At minimum, add a length-normalised likelihood and a shortest-correct-trace heuristic to Tables 3 and 10 to isolate whether LALP's advantage is locality-specific or is really about avoiding GALP's verbosity/self-conditioning bias. This addresses Locality is claimed as causal but shown only via correlation and Baseline set is narrow.

Suggested

• Clarify the Table 22 identical-row observation. Either explicitly state that the LALP selection set is unchanged under the two additional teachers (and quantify how often the added teachers would have won a per-prompt argmax), or rerun that row so the numbers reflect the enlarged pool. This addresses Table 22 LALP row is bit-identical to Table 21.
• Re-scope the GPQA-D SOTA claim. Remove or explicitly qualify "outperforms other state-of-the-art models" — LIMO-32B, Sky-T1, and OpenThinker2 differ in training data, size, teacher mix, and epochs, so the comparison is not a controlled test of the data-curation contribution.
• Standardise notation and cross-references. Pick either $k$ or $\alpha$ and give the mapping; replace "9.4%" with "+9.4 points" throughout; fix the Table 2 vs Table 9 cross-reference for the 0.719 value; fix the Qwen2.5-72B name/URL inconsistency; correct the Table 8 duplicated heading; either extend Table 19 to both students or fix its caption; note in the Table 20 caption that AVG excludes GPQA. These are the Notation and cross-reference slips items.
• Report LALP preprocessing wall-clock cost. Replace the qualitative "manageable" with concrete numbers (per-response seconds, per-teacher total, GPU-hours for the LIMO pool) so future users can budget.
• Soften "consistently … by a large margin" in the abstract to reflect that LCB-easy regresses relative to the base model and LCB-hard is +2.9 points.
• Extend the Llama-3.1-8B student ablation to include the two headline claims (teacher-ranking recovery and mixed-pool selection) so the paper's "three student models" framing is honoured beyond a single appendix table.