Procedural Pretraining: Warming Up Language Models with Abstract Data

Summary

The paper repositions procedural data — outputs of simple algorithms such as Dyck languages, elementary cellular automata, sorting, and stack simulations — as a lightweight warm-up stage for language-model pretraining rather than a substitute for it. The conceptual move is to treat semantic and algorithmic content as separable ingredients: procedural tokens supplied before natural language, code, or informal math give the model a scaffold for symbol manipulation, long-range recall, and compositionality that standard corpora impart less cleanly. Three claims are backed empirically: (i) different procedural tasks improve different diagnostic skills (Dyck for needle-in-a-haystack recall, ECA Rule 110 for reversed addition, SORT/UNION for sorting), and simplistic alternative explanations (attention sharpening, weight rescaling) fail to reproduce the gains; (ii) 2–4M procedural tokens (0.1–0.3% of the semantic corpus) reduce C4, CodeParrot, and DeepMind-Math validation losses and can substitute for 14–45% of semantic tokens; (iii) attention layers carry the transferable structure for pure code while MLPs do so for natural language, hinting at a mechanistic separation of roles. Section 6 sketches how data mixtures and per-component weight assembly can combine multiple procedural sources. The empirical scope (multiple corpora, models up to 1.3B, per-component transfer, downstream tasks) is genuinely broad and the ablations are thoughtful.

Two conceptual limitations bound the contribution. First, the T_1=0 baseline does not control for the extra compute added by the warm-up, and there is no head-to-head against a comparable warm-up on unstructured text or code — leaving open whether the gains flow from procedural structure specifically or from any structured warm-up. Second, the selection of procedural tasks and transfer modes is done post-hoc across many combinations without correction for multiple comparisons or seed variance at scale, so several headline effects (Table 3 point estimates, GLUE per-task jumps, MBPP 1–3% pass@k) are hard to calibrate. The framing around 'disentangling knowledge from reasoning' is aspirational rather than measured.

Strengths

• Conceptual contribution. Reframing procedural data as a complement to — rather than substitute for — standard pretraining, and grounding it in a suite of algorithmic diagnostics (haystack, addition, sorting) that make the mechanism claims concrete, is a clean and useful move.
• Breadth of ablations. The negative controls (attention-sharpening regularizer, per-layer weight shuffling, Gaussian noise injection) are the right ones to run and collectively rule out the most common 'it's just optimization' rebuttals.
• Component-level localization. The finding that attention layers dominate for pure code and MLPs dominate for natural language is non-obvious and, if replicated at scale, is genuinely mechanistically interesting given the standard 'MLPs = knowledge' framing.
• Scaling evidence. Extending the additive and substitutive settings to 350M and 1.3B parameters (Appendix L) partially addresses the usual small-model criticism of synthetic-pretraining papers.
• Constructive combination. Section 6's proof-of-concept that data mixtures and per-component weight mixtures can beat single-source procedural warm-ups is a genuinely new direction for the sub-area.

Weaknesses

• Compute-matched baseline missing. The T_1=0 baseline sees strictly less training compute than the warmed-up model, and the paper's own Section 5.1 observation that gains are 'not clearly correlated with' T_1 is compatible with a generic 'more training helps' explanation. Without an equal-compute control (e.g. matched-token warm-up on unstructured text or code, or equivalent extra semantic-token steps), the claim that structure is the active ingredient is under-supported.
• No head-to-head against structured warm-ups from prior work. The paper positions procedural warm-up against 'standard pretraining' but not against the natural alternatives cited (Hu et al., 2025 formal-language warm-up; Aryabumi et al., 2024 / Petty et al., 2024 code as pretraining). Since these are precisely the closest prior baselines, their omission weakens the practical significance of the improvements.
• Multiplicity and cherry-picking. Ten procedural tasks × several downstream domains gives many degrees of freedom, and the paper carries forward only the best per domain (SORT / UNION / SET) into headline claims without an explicit multiple-testing accounting or a rule for how the 'best' would have been chosen prospectively. This particularly affects the mixture and weight-mixture claims in Section 6.
• Scaling table has no error bars. Table 3's differences at 350M (40.3 vs 39.0 C4; 4.97 vs 4.62 CODEPARROT) and 1.3B (28.8 vs 27.3; 3.45 vs 3.36) are single-seed point estimates. Given that the paper argues elsewhere for seed replication as central methodology, the same care at scale is needed to distinguish real improvement from run-to-run drift.
• Downstream evidence is thin. Figure 7 highlights an MBPP pass@k of roughly 1–3% as the flagship downstream result and the take-away then claims persistence 'after downstream fine-tuning' — but Figure 7 is few-shot, and Appendix M's GLUE gains lean heavily on two very large single-task swings (STS-B +13.7, WNLI +9.8) with no per-seed variance.
• Bimodal seed distributions summarized as mean±std. SET (full transfer) HAYSTACK 18.9±26.6, STACK (full) HAYSTACK 55.2±39.3, and similar entries indicate strongly skewed / bimodal outcomes over seeds. Reporting median and success rate would better convey what the tables actually measure.
• Take-away claims are stronger than their tables support. Several summary sentences overreach: (a) 'attention layers support structured domains such as code and mathematics' is contradicted by the same section's finding that full-model transfer wins on documented code and informal math; (b) 'effects are additive' is contradicted by SET(attn)+ECA(MLP) losing to single-source models on 2 of 4 tasks; (c) 'low entropy is a side effect not the cause' is a strong causal claim from a single soft-penalty regularization control.
• Numerical inconsistencies. 10.5B vs 10B tokens for the same Appendix-L scaling study; the substitutive '66% of C4' and '2.1B token reduction' do not both hold against 6.6B C4 (they imply 68.2% and 2.24B respectively); the 2× input-length token-counting convention conflicts with the 'output ≤ 2× input' description; the Section-4 curriculum vs Section-5 fixed-length regime is not reconciled.
• Underspecified methodology. k-DYCK generation policy at length 128 (rejection vs concatenation is not stated); STACK 'uniqueness' rule (globally unique tokens vs currently-unique-on-stack); Gaussian-noise σ scale (absolute vs per-layer scaled) — each is enough to prevent a faithful reimplementation.
• Framing overreach. The abstract's 'disentangling knowledge acquisition from reasoning' promise, the Discussion's appeal to 'small Kolmogorov complexity' of procedural data, and the developmental-analogy motivation are decorative rather than measured; they should be labelled as conjectures.

Reproducibility & code

• ECA Rule 110 generator missing. ECA is a marquee data type — it is the 'BEST' bar for REVERSED ADDITION in Figure 2, appears in Tables 8, 11, 12, 13, and is one of the two ingredients of the Section 6.2 / Table 1 weight mixture. The released procedural_data/ directory has nine generators but no ECA module and no entry for it in TASK_TO_MODULE.
• SORTING downstream task missing. SORTING anchors a column of Figure 2 and of Tables 8, 12, 13, 14. The downstream/algorithmic_tasks/ directory has no task_sort.py; the README lists only four tasks. Table 12 and the SORTING columns elsewhere are not reproducible from artefacts.
• General procedural-sequence shuffling absent for non-Dyck tasks. The 'Best model shuffled' control (Figure 2) requires per-task shuffling; only dyck_shuffle.py is shipped. Every non-Dyck bar in that control is unreproducible.
• Perturbation controls (weight shuffle, Gaussian noise, entropy regularizer) not implemented. Appendix F's negative controls — layer-wise weight shuffle (Figure 13, Table 13), Gaussian noise injection at σ ∈ {0.01, 0.05, 0.10} (Figure 14, Table 13), and the entropy regularizer targeting τ=0.8 on three heads (Figure 12) — have no counterpart in the shipped code. No state-dict shuffler, no noise-injection utility, no entropy loss term, no per-head selection helper.
• WikiText, JavaCorpus, BLiMP pipelines not shipped. Figure 4 (Section 5.1), Figure 6 top row (Section 5.4), Figures 21–22, Section 6.1 mixtures, and Appendix J vocabulary sweep all depend on WikiText and/or JavaCorpus training loops and a BLiMP evaluator. Grep returns zero matches for any of these in the released repo.
• Downstream evaluators for Section 5.3 / Appendix M absent. MBPP (pass@k, few-shot), GLUE, PY150 code completion, WikiText-103 fine-tuning, ARC-Easy and HellaSwag zero-shot — all six evaluators referenced in the downstream-tasks section — have no loaders, no fine-tuning scripts, and no evaluation harness in the release. Every headline downstream number is unreproducible from artefacts.
• 350M / 1.3B scaling infrastructure absent. SIZE_TO_CONFIG has no aliases for 350M / 1.3B; there are no C4 or CodeParrot configs at these sizes; no Pythia-suite architecture wiring is present. Table 3, Table 4 (BLiMP) and the 82M-procedural-token substitutive result at 1.3B all sit on this missing scaffolding.
• Per-head selective transfer and component-wise weight merging absent. COMPONENT_PATTERNS in downstream/utils.py slices only at 'attn'/'ffn'/'ln' granularity, so the three-heads experiment in Appendix F.1.2 has no reproducible implementation. load_pretrained_weights loads from a single pretrained_path, so the Section-6.2 SET-attention + ECA-MLP composition has no shipped merging utility.
• Data-mixture pretraining (Section 6.1) not wired. The source-prefix token scheme and T_A/T_B ratio sweeps are not implemented in procedural_pretraining/utils.py or procedural_data/*.
• No shipped configs for the specific procedural-token budgets. The end-to-end C4 / CodeParrot / DeepMind-Math scripts exist and accept a pretrained checkpoint, but no config files fix the 2.1M / 10.4M / 82M procedural-token budgets that anchor every headline number, and no baseline curves are checked in — so the 55/67/86% substitutive percentages must be regenerated from many additional sweeps.
• What the release does cover. Core procedural pretraining for nine generators, the three semantic pretraining scripts (C4, CODEPARROT, DEEPMIND-MATH), full-model / attn-only / MLP-only transfer at the layer level, and dataset-download shell scripts for C4 and DeepMind-Math. This is a substantial fraction of the pretraining half of the paper, but essentially none of the downstream evaluation or perturbation-ablation half.

Recommended Changes

Essential

• Add a compute-matched baseline. For at least one flagship dataset (e.g. C4), run either an equal-token warm-up on a chunk of unstructured text or code, or extra continued semantic pretraining that matches the T_1 compute of the procedural warm-up, so the reader can see the structure-specific gain net of the compute confound raised in Weaknesses.
• Head-to-head against Hu et al. 2025 formal-language warm-up and code-as-warm-up. These are the closest prior recipes and are already cited; a single row comparing procedural warm-up to a formal-language warm-up (Hu et al.) and a small-code warm-up (Aryabumi/Petty) at matched tokens would greatly sharpen the paper's positioning.
• Report seed variance for the scaling results. Add at least a small-N (e.g. 3-seed) run for Table 3 and Table 4, and mark which perplexity gaps survive noise — particularly the 3.45→3.36 CODEPARROT number at 1.3B.
• Reconcile the substitutive percentages with the token counts. The 66% / 75% figures and the 2.1B / 2.5B token reductions do not both hold against 6.6B C4 / 10.5B CODEPARROT; correct or clarify. Also align 10B vs 10.5B tokens across abstract, introduction, and Section 5.2.
• Ship the missing code paths. Provide ECA Rule 110 generator, SORTING downstream, general procedural-sequence shuffling, layer-wise weight shuffle, Gaussian noise injection, entropy regularizer, per-head selective transfer, component-wise weight merging, source-prefix-token mixture harness, WikiText / JavaCorpus / BLiMP pipelines, and the Section-5.3 downstream evaluators (or clearly document why they are omitted). At minimum, list which experiments the release does not cover in the availability statement.
• Release the specific configs. Ship YAML presets at the T_1 = 2.1M / 10.4M / 82M procedural-token budgets and the 350M / 1.3B model sizes, plus the baseline-curve reference numbers used to derive the 55/67/86% substitutive percentages.

Suggested

• Rephrase overreaching take-aways. (a) Localization take-away: change 'attention layers support structured domains such as code and mathematics' to reflect that full-model transfer is best on documented code and informal math. (b) Combination take-away: replace 'additive' with 'complementary on average / no severe weakness on any task' and note the per-task regressions in Table 1 / Table 14. (c) Downstream take-away: reframe to '... in few-shot and fine-tuning settings' to match Figure 7. (d) Attention-sharpening: soften 'not the cause' to 'sharpness is necessary but not sufficient'.
• Report medians and success rates for bimodal seed distributions. Where the reported std spans a large fraction of the [0, 100] range (e.g. SET/STACK full-transfer HAYSTACK; 16-DYCK SHUFFLE HAYSTACK), also give a median and a per-seed success bar.
• Address multiple-testing / selection bias. Add a section explaining how the 'best procedural task per domain' is chosen and either report all tasks per domain, or account for the selection in the reported effect sizes.
• Discuss the 4-DYCK HAYSTACK partial-transfer collapse. Explain in-text why full-model 4-DYCK excels while neither attention-only nor MLP-only does — this is a striking exception to the localization narrative and belongs in Section 4.2, not left tacit in Appendix N.1.
• Fix or clarify numerical / methodological ambiguities. Reconcile the '2× input length' token accounting with 'output ≤ 2× input'; specify k-DYCK length-128 generation policy; disambiguate STACK 'uniqueness'; state whether Gaussian noise σ is absolute or per-layer scaled; state whether Section-4 curriculum vs Section-5 fixed-length changes results.
• Soften decorative claims. Reframe 'disentangling knowledge from reasoning', 'small Kolmogorov complexity', and the humans/infants analogy as conjectures / motivating intuitions rather than technical claims, and drop the '10 vs 10.5B' and 'SORT-vs-UNION best-on-C4' inconsistencies with a single-line explanation of scale-dependent task choice.
• Disambiguate the two 'shuffled' labels. Rename either the Table 8 'BEST MODEL SHUFFLED' (data-level) or the Table 13 'Shuffled' (weight-level) row to prevent conflation.