Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Summary

The paper introduces prescriptive scaling: given a base-model pre-training FLOPs budget, what post-training accuracy can one realistically expect on a target benchmark? Rather than modeling mean trends of downstream performance, the authors estimate high conditional quantiles ("capability boundaries") using smoothed pinball regression at $\tau=0.98$ with a monotone-saturating four-parameter sigmoid in log-compute. Fits are performed on ~5k Open LLM Leaderboard v2 checkpoints and a newly evaluated ~2.4k open-weight set (PROTEUS-2K), across six benchmarks and four chronological periods from mid-2022 to March 2026. The central conceptual move — trading individual model prediction for an envelope over a heterogeneous population of post-trained artifacts — is well motivated: quantile boundaries are more robust than sample maxima, coverage becomes a natural diagnostic for temporal drift, and a four-parameter sigmoid captures the observed compute-vs-attainable shape as well as more flexible I-spline estimators on most tasks, while cleanly exposing math reasoning as a task whose boundary keeps advancing. The paper builds a useful stack of companion results around this: a task-dependent size–time saturation diagnostic, a cross-benchmark shift test for AIME-2025 contamination, external-validity overlays on frontier and newly-evaluated models, and a bin-balanced I-optimal design that recovers near-full-data boundaries at roughly 20% of parameter-count-weighted evaluation cost (as little as 5% on some tasks). Independent replication of the shipped code reproduces the headline pipeline cleanly (Table 1 to ~0.001 per cell, all 16 Table 2 cells within tolerance, Table 3 exact, the contamination-test p-value at 0.148 vs. 0.15), which increases confidence in the empirical picture. The main conceptual limitation is a persistent tension between Definition 2.1 — a generative/engineering statement about training a model to reach $y$ — and the estimator, a descriptive quantile of an observed post-trained population subject to strong selection effects and prevailing recipes. Several derivative claims (the boundary "reliably upper-bounds" attainable accuracies; "no clear evidence" of AIME-2025 contamination from $p=0.15$; a two-parameter logit-linear form for the cross-benchmark test that does not follow from the paper's own four-parameter sigmoid) reach further than the estimator supports, and Table 2 carries no uncertainty quantification.

Strengths

• Conceptual reframing. Recasting downstream scaling as a high-quantile envelope over post-trained checkpoints — rather than a mean or a sample maximum — cleanly resolves the outlier and heterogeneity problems that have plagued downstream scaling analyses. Section A.2 and the illustrative qingy2024/Benchmaxx example (verifiable directly from the shipped leaderboard CSV) make the case for quantiles vs. maxima crisply.
• Temporal validation protocol. The rolling fit-on-$P_t$ / evaluate-on-$P_{t+1}$ design, combined with signed coverage error and pinball loss, gives concrete purchase on whether a compute-only boundary transfers across LM generations. Restricting OOD evaluation to the $z$-overlap avoids extrapolation and is done consistently.
• Function-class comparison and simplicity. The head-to-head against constant, binwise, and monotone I-spline estimators — together with the $(\kappa, \lambda)$ sensitivity sweep — justifies the four-parameter sigmoid without hand-waving. Finding that a strictly more flexible I-spline does not improve OOD calibration is a useful negative result, and the sensitivity sweep confirms in-period cross-validation does not change the conclusions.
• Balanced I-optimal design. Adapting I-optimality with a bin-balance term and a knapsack-style greedy solver, then showing 20% (5% on some tasks) of parameter-count-weighted evaluation cost recovers full-data frontiers, is a useful practical contribution independent of the sigmoid framing.
• Artifact release. PROTEUS-2K materially extends the leaderboard corpus into the post-retirement regime and covers new base families (Qwen3, Gemma-3, GPT-OSS, OLMo-3, Nemotron nano/cascade); the shipped proteus_2k.csv matches the ~2.4k count.
• Reproducible headline numbers. Independent execution of the shipped code reproduces the Table 1 attainable accuracies to ~0.001 per cell, all 16 Table 2 pinball/calibration cells within tolerance, all 8 Table 3 size–time statistics exactly, the pretrained-vs-post-trained adjacent-decrease heatmap exactly, the PCA scaling qualitatively, and the contamination-test p-value at 0.148 vs. the paper's 0.15. The core empirical picture holds up under execution.
• Honest surfacing of the math-reasoning exception. MATH Lvl 5 is repeatedly flagged as the case where the boundary advances over time, giving a clean example of what a shifted boundary looks like in this framework rather than being buried.

Weaknesses

• Prescriptive definition vs. observational estimator. Definition 2.1 frames the object as "train from scratch a model end-to-end to achieve a targeted downstream performance $y$", but the paper never trains a model — it fits a conditional quantile of the score distribution over an existing corpus. The forward-looking claims about what a practitioner can "reliably reach" depend on treating that empirical envelope as prescriptive when it is fundamentally descriptive and conditional on the observed recipes and post-training methodologies. The estimator-vs-definition tension is not reconciled.
• "Reliably upper-bound" is inconsistent with quantile semantics. Section F concludes the fitted boundary "reliably upper-bound[s] the best possible accuracies attainable" with MATH Lvl 5 outliers as "one caveat"; but a 0.98-quantile expects ~2% of models to lie above by construction, and this is exactly the argument Section A.2 uses to defend using a quantile over a conditional maximum. The two sections read against each other.
• Cross-benchmark shift test rests on a step that does not follow. The test assumes $\mathrm{logit}(y_1)$ is approximately linear in $\mathrm{logit}(y_2)$ because both are sigmoid in $z$. That is true for the two-parameter sigmoid $\mathrm{logit}(y) = a + \beta z$, but not for the paper's four-parameter form $q_\tau = y_0 + L\,\sigma(a+\beta z)$: adding $y_0, L$ breaks the clean logit-affine relationship. The identifying assumption for Section 5.2.2 is therefore inconsistent with the estimator used elsewhere.
• "No clear evidence" of AIME-2025 contamination from a $p=0.15$ null. The contamination case study reports a positive point estimate for $\gamma$ (replication: $\hat\gamma \approx 0.58$, se $\approx 0.33$, one-sided p $\approx 0.148$) at $n=90$, then concludes "no clear aggregate evidence" both in the abstract and in the findings. This is an underpowered null result, not evidence of absence. There is no power calculation, no interval estimate for $\gamma$, and no minimum-detectable-effect analysis.
• No uncertainty on Table 2 aggregates. Table 2 is the workhorse comparison across four estimator families, averaged over $3 \times 6 = 18$ underlying values per cell, but no standard errors, per-split ranges, or paired tests are reported. The relative ordering the paper leans on differs by ~10% relative and could easily be within noise.
• Table 2 layout hides the mapping between ID/OOD and estimator. ID and OOD values are interleaved inside a single cell per column, so verifying the headline "best OOD calibration" claim requires reverse-engineering the row order.
• Balanced I-optimal ablation is missing. Finding 4 mixes standard I-optimality ($\lambda=0$) with the bin-balance term ($\lambda>0$), and Figure 5 shows only the combined design. The 20% (5% on some tasks) budget claim could reflect the informative design, the balance term forcing coverage, or simply a stratified-random baseline at matched cost — no ablation isolates the contributions.
• Non-stationarity of MATH is framed as a strength rather than as a limitation of the framework. Under-coverage of $\sim -3\%$ to $-5\%$ on MATH across three rolling splits, plus the boundary crossings visible in Figure 4 on newly released models, is exactly the case where prescriptive forecasting fails. This should be presented as a scope limitation, not a diagnostic success.
• Compute-only conditioning ignores post-training compute. The boundary is a function of base-model pre-training compute only, yet the boundary describes performance after post-training pipelines whose compute has itself been scaling — precisely for MATH Lvl 5. Adding even a coarse post-training-compute proxy could absorb some of the temporal drift.
• Two different "2%" quantities are used interchangeably. The abstract's "coverage error below 2%", Figure 2's "$\pm$2% signed coverage", and the "2.2% vs 3.6%" contribution bullet (an average absolute calibration error) refer to three related-but-distinct quantities. The near-identical numbers invite conflation.
• "Single latent component" claim skips the variance-explained step. Appendix H.3 concludes the observed benchmark scaling is "largely attributed to advances in a single component" from three PC scaling plots, but does not report PC1's variance share — flatness of PC2/PC3 in compute is not sufficient (replication: PC1 EV $\approx 0.77$, top three $\approx 0.94$; the paper does not surface these).
• Size–time boundary model equation is missing. The manuscript announces "We model $\mathrm{logit}(q_\tau^b(x,t))$ as" and then omits the display equation; the functional form of the time trend $\phi_b(\cdot)$ and the fitting procedure are never specified. Table 3 is therefore not reproducible from the paper alone.
• 13B saturation threshold is asserted but not justified. The saturation diagnostic and the size–time model both split at 13B, and the narrative ("MMLU-Pro is scale-limited; MATH Lvl 5 is not") is entirely relative to that boundary. No sensitivity across thresholds is provided.
• Compute-conditioning pipeline is opaque. The independent variable is "estimated FLOPs" for thousands of open-weight checkpoints and closed-source frontier models, but the manuscript never describes how $C_i$ is computed, how measurement error is quantified, or how MoE / active-vs-total-parameter distinctions are handled — despite this being the primary conditioning axis.
• External-validity check does not partition by family. The PROTEUS-2K held-out set is dominated by base families that already appear in the training corpus (Mistral-7B, Qwen2.5, Llama-3.2, gemma-2). Whether the boundary would still cover a set with different family composition (the OLMo-3 / Nemotron / GPT-OSS / Qwen3-4B-Base subsets alone) is not tested.
• Selection caveat is present but not carried into the headline claims. The paper flags that interpretation is relative to the observed model population, but abstract-level statements ("reliably upper-bound", "attainable") do not carry that caveat. Table D.1 shows heavy concentration on Llama, Qwen, Mistral, and Gemma bases.
• v1 saturation interpreted causally. Appendix H.2 concludes v1 tasks were "too old or faced severe contamination issues" from saturation curves alone; Appendix I explicitly cautions the plots are observational. The two passages contradict each other.
• Notation collision on $\lambda$. Section 2.1's ridge weight ($\lambda=10^{-3}$), Section 4.1's balance/trade-off ($\lambda \geq 0$), and Appendix J's ridge prior ($\eta=10^{-9}$) all appear in overlapping contexts with overlapping symbols.
• Regression equation vs. figure caption mismatch. Equation (3) uses $\mathrm{logit}(0.01\, y_i)$, but Figure 7(b) reports its fit as $\mathrm{logit}(100\, \hat{y})$ while the axis labels use $\times 0.01$.
• Appendix E mis-attributes classical scaling laws. The sentence "This is in contrast with classical scaling laws (Kaplan et al., 2020) use..." is ungrammatical and attributes downstream-task prediction to Kaplan et al. — the very framing the related-work section pushes back against.

Reproducibility & code

• Headline aggregates reproduce cleanly. Executing the shipped pipeline reproduces the sigmoid vs. constant OOD calibration comparison at $0.0221 / 0.0360$ (paper: $2.2\%$ / $3.6\%$), and all four estimators land within tolerance on both pinball and calibration ID/OOD. The temporal-drift diagnostic reproduces the "four of six tasks below 2%, MATH Lvl 5 and IFEval under-covered" pattern with signed coverage errors matching the paper's ranges.
• Table 1 reproduces to $\sim 10^{-3}$ per cell by interpolating the no-split sigmoid fit at $z=24$, but regenerate_paper_artifacts_from_scratch.sh never writes a table1_attainable_accuracies_1e24.csv; the six headline numbers must be interpolated by hand from the fitted curves.
• Table 2 reproduces but is spread across two subtrees. Pinball baselines for the four estimators live in scripts/evaluate/sigmoid_pinball_baselines.py; bin-wise coverage/calibration lives in scripts/evaluate/sigmoid_binned_mae.py under a different output tree, and only the Sigmoid row currently receives calibration error from a single script. Assembling the $4 \times 4$ table required extending the baselines script and joining across trees.
• Table 3 reproduces exactly but is not part of the regenerate pipeline. Running open_llm_leaderboard_causal_size_effect.py --tau 0.98 --size_threshold_b 13.0 reproduces all eight cells, but the script is not called by the top-level driver and the late-period cutoff is a CLI argument rather than a value pinned to the paper.
• Contamination test (Section 5.2.2) reproduces but is not automated. plot_math500_vs_aime2025.py:_contamination_test_logit_to_logit yields $n=90$, $\hat\gamma \approx 0.58$, one-sided p $\approx 0.148$ (paper: 0.15), but the script is not invoked by the regenerate driver, and the "overlapping range" definition that yields $n=90$ is a numeric threshold embedded in the script rather than a documented protocol.
• Frontier fit (Figure 23) reproduces qualitatively; MMLU/MMLU-Pro magnitudes drift. Re-running the shipped frontier scripts reproduces the near-equality of sigmoid and I-spline losses on every task, and AIME 2025 matches to $10^{-4}$ (0.0087/0.0088 vs. paper 0.0087/0.0088). MMLU (0.0051/0.0052 vs. paper 0.0042/0.0043) and MMLU-Pro (0.0050/0.0051 vs. paper 0.0045/0.0045) come out ~11–21% higher, most likely because tables/LifeArchitect.xlsx is a continuously-updated live source that has accumulated more model rows since the paper snapshot. The qualitative claim survives; the absolute magnitudes are not bit-reproducible.
• PCA (Appendix H.3) reproduces qualitatively but is not automated. scripts/run/latent_pca_sigmoid.py gives top-three cumulative variance in the 90.3%–93.7% range (nearest to the paper's 95% for P3→P4 at 93.7%), with PC1 dominant at $\approx 0.77$. The script is not invoked by the regenerate driver, and PC1's variance share is never reported in the manuscript.
• $(\kappa, \lambda)$ sensitivity sweeps (Figures 20–22) are not automated. Both sensitivity_kappa_lambda.py and sweep_kappa_lambda.py encode the correct fit-on-$P_k$ / evaluate-on-$P_{k+1}$ protocol and confirm the paper's conclusions (CV-selected $\kappa \in \{20, 50\}$, $\lambda \in \{10^{-4}, 10^{-3}\}$; essentially zero OOD improvement over the fixed default), but neither is called by the top-level driver, so the whole "fixed defaults are not artifacts" argument cannot be reproduced end-to-end.
• What is well-covered. The core sigmoid + rolling-split OOD coverage pipeline, the pretrain-vs-post-train comparison and monotonicity-violation rate, the balanced I-optimal $\alpha$ sweep, the Proteus-2k overlay, and the GPQA-diamond frontier fit are all directly wired into regenerate_paper_artifacts_from_scratch.sh and their upstream data ships.

Recommended Changes

Essential

• Reconcile Definition 2.1 with the estimator. Rewrite Definition 2.1 as a distributional/attainability statement about the observed post-trained population, or explicitly identify the conditions (family coverage, recipe uniformity, no post-hoc selection) under which the observed quantile can be read prescriptively. Carry the wording change through the abstract, findings boxes, and Section 5.
• Fix the cross-benchmark shift-test derivation. Either restrict Section 5.2.2 to the two-parameter sigmoid $\mathrm{logit}(y) = a + \beta z$, or derive the correct implied relationship between $\mathrm{logit}(y_1)$ and $\mathrm{logit}(y_2)$ when both boundaries have the four-parameter form $y_0 + L\,\sigma(a+\beta z)$. Re-run the OLS regression under the corrected specification.
• Recast the AIME-2025 contamination conclusion. Add an interval estimate for $\gamma$ in Eq. (3) and a power calculation or minimum-detectable-effect analysis; replace "no clear evidence" with an explicit "we failed to detect inflation at $\alpha=0.05$; effect sizes up to $X$ pp are not rejected" and update the abstract accordingly.
• Restructure Table 2 and add uncertainty. Break out ID and OOD into separate columns per metric, give each estimator its own row, and report per-split (and ideally per-task) variability (bootstrap SE or IQR across the 18 underlying values). Add a paired test for the "sigmoid best OOD calibration" claim.
• Restore the missing size–time equation and specify $\phi_b(t)$. Add the display equation for $\mathrm{logit}(q_\tau^b(x,t))$, specify $\phi_b(\cdot)$'s functional form and knots, and state at which $t$ Table 3's $\hat q_{0.98}(13\mathrm{B}, g{=}1)$ is evaluated, so the $0.03 \to 0.94$ shift for MATH Lvl 5 can be decomposed into the $g$-indicator step and the smooth trend.
• Ship a driver that regenerates Table 1, Table 3, Section 5.2.2, and Appendix H.3. Wire open_llm_leaderboard_causal_size_effect.py, plot_math500_vs_aime2025.py, latent_pca_sigmoid.py, and a Table-1-CSV writer into regenerate_paper_artifacts_from_scratch.sh; extend sigmoid_pinball_baselines.py (or add a joined driver) so all four estimators emit both pinball loss and coverage/calibration error in a single Table 2-shaped CSV. Also automate the $(\kappa, \lambda)$ sweeps used to defend the fixed defaults.
• Freeze the frontier-model input. Snapshot tables/LifeArchitect.xlsx (or ship a dated CSV) so Figure 23's absolute MMLU/MMLU-Pro pinball losses are bit-reproducible rather than drifting with a live spreadsheet.
• Add a bin-balance / I-optimality / stratified-random ablation to Figure 5. At minimum include the $\lambda=0$ curve and a stratified-random baseline at matched cost so Finding 4's "20% budget" claim can be attributed to the design rather than to coverage alone.

Suggested

• Reframe MATH Lvl 5 non-stationarity as a scope limitation rather than as a diagnostic success, and note in the abstract that prescriptive forecasting for reasoning tasks is currently tracking a moving target.
• Rephrase "reliably upper-bound" throughout the manuscript to a statement about the 0.98 quantile dominating the great majority of new observations, and treat MATH Lvl 5 as an example of a shifted boundary (as Remark 3.1 already implies) rather than an "exception" to an upper-bounding claim.
• Disentangle the three "2%" numbers. Label each occurrence as either per-task signed coverage error (Figure 2's $\pm 2\%$), abstract-level per-task absolute error on the four stable tasks, or averaged OOD calibration across all six tasks and periods.
• Report PC1's variance share (and per-benchmark loadings) alongside the "single dominant latent axis" claim in Section 3.2 / Appendix H.3.
• Ablate the 13B threshold. Report the Table 3 statistics and Figure 6-style saturation curves at $\{7\mathrm{B}, 13\mathrm{B}, 20\mathrm{B}, 30\mathrm{B}\}$ so the "MMLU-Pro remains scale-limited" narrative is not tied to one cutoff.
• Describe the FLOPs-estimation pipeline for base checkpoints and closed-source frontier models — sources, formulas (e.g., $6ND$), handling of MoE active-vs-total parameters — and, ideally, a robustness check that shifts $z_i$ by the estimated measurement error.
• Add a coarse post-training-compute proxy (or at least a categorical fine-tuning-vs-RL flag) as a secondary conditioning variable so the MATH non-stationarity can be partly absorbed into methodology.
• Break out external-validity by family on Proteus-2k (leaderboard-fit vs. OLMo-3 / Nemotron / GPT-OSS / Qwen3 subsets alone) so the "boundary continues to upper-bound" claim is not driven by base-family overlap with the training corpus.
• Rename $\lambda$ in one of the three contexts (ridge in Section 2.1, balance in Section 4.1, ridge prior in Appendix J) so the three regularization-flavored constants are not collided.
• Fix Figure 7(b) to use $\mathrm{logit}(0.01\, \hat{y})$ (matching Eq. (3) and the axis labels), and correct the "This is in contrast with classical scaling laws (Kaplan et al., 2020) use..." sentence in Appendix E — both for grammar and to avoid attributing downstream-task prediction to classical loss-based scaling laws.
• Soften Appendix H.2 on v1 contamination so the "strongly indicates ... severe contamination" wording matches Appendix I's careful "these plots are observational" caution, or supplement with an independent contamination test on v1 that can distinguish saturation from leakage.