How much can language models memorize?

SAI replication review · Referee report

Summary

The paper proposes an information-theoretic definition of memorization that separates unintended memorization (bits about a specific dataset) from generalization (bits about the data-generating process), building on Kolmogorov complexity and operationalising it via model likelihoods. The central conceptual move is to eliminate generalization by construction — train on uniformly-random bitstrings, where Proposition 6 forces generalization to zero — and thereby measure raw capacity in bits. Doing so at scale across an $\{n_\mathrm{layer}\}\times\{d_\mathrm{model}\}\times\{\text{precision}\}$ grid yields a headline of roughly $3.5$ – $3.8$ bits per parameter for GPT-style transformers, with the precision uplift from bf16→fp32 far smaller than the 2× uplift in raw parameter bits. The framework is then re-applied to real text (FineWeb) with a large oracle reference, giving a clean story for double descent (it starts when dataset-in-bits exceeds capacity-in-bits) and a sigmoidal scaling law for membership-inference F1 validated out-of-sample on GPT-2 XL (1.5B). The extraction analysis is a genuine contribution: on well-deduplicated data, the training-set extraction rate converges to the test-set rate at large $N$ , a sharp argument that greedy-extraction has been over-interpreted as evidence of memorization. Two main conceptual limitations remain: (i) an unavoidable dependence of MEM $_U$ on the reference model — the A.13 ablation is honest about a 2.5× swing across a 6× reference-size range, so absolute bit values only carry a relative interpretation — and (ii) capacity is measured under Adam-SGD and is therefore a lower bound whose looseness is not quantified. An executed reimplementation from the manuscript confirmed the method is faithful to the paper's description and reproduced its parameter counts to within 0.1% and its qualitative predictions (memorization plateau in one saturated cell, monotone F1 decay in dataset size) where compute allowed, but the exact 3.6-bpp headline was not reachable inside the reproducer's budget — the one saturated bf16 cell landed at 2.76 bpp, systematically below rather than around 3.6. There are also notational and consistency issues throughout (multiple headline bpp numbers, an F1 accuracy claim Table 2 does not support, a Table 2 label mismatch, and a formula that writes a likelihood where a code length is meant). Fixed, this is a substantive contribution to how the community should think about, and measure, LM memorization.

Strengths

Conceptual contribution. The unintended/intended decomposition is the right way to separate 'the model knows this because of the sample' from 'the model knows this from the population', and the compression-based operationalisation avoids the extraction-vs-generation confounds discussed in Section A.3.
Calibration via synthetic bitstrings. Uniform-random data has analytically known entropy $NS\log_2 V$ , so capacity is measured as the gap to a known reference and is not merely a re-parameterisation of Kaplan-style held-out curves; the Section 3.2 'Relationship to scaling laws' discussion makes this defensible.
Empirical scope. The synthetic sweep spans 16 $(n_\mathrm{layer}, d_\mathrm{model})$ cells × 2 precisions × many dataset sizes × 5 seeds; text experiments cover four target sizes and eight dataset scales; A.13 varies reference-model size across three points. This is unusually thorough for a memorization paper.
Out-of-sample scaling-law validation. Training a fresh GPT-2 XL (1.5B) at three predicted F1 targets and hitting the endpoints within ~1 point is real external validation rather than in-sample fit — it distinguishes this scaling law from most in the literature.
Extraction/membership decoupling. The observation that training-set extraction rate collapses onto the test-set extraction rate at large $N$ under strict deduplication is a genuinely useful methodological correction to the community's over-reliance on extraction as evidence of memorization.
Honest reference-model ablation. A.13 does not hide the 2.5× swing in absolute MEM $_U$ across a 6× reference-size range and explicitly names the ref≈target regime as a limitation.
Method is learnable from the manuscript. An independent reimplementation drawn from the paper alone matched the paper's parameter counts to within 0.1% and reproduced qualitative behaviours where saturation was reached (one bf16 cell shows a clean memorization plateau; membership F1 falls monotonically with dataset size across every configuration). The description of the pipeline is precise enough that a careful reader can recover the load-bearing pieces (analytical $NS\log_2 V$ entropy, the clipped max $L_\theta - L_{\hat\theta}$ MEM $_U$ , and the balanced-pool median-threshold F1) without ambiguity.

Weaknesses

Multiple, inconsistent headline bpp numbers. The bf16 estimate is stated as 'approximately 3.6' (abstract), '3.5–3.6' (Section 3.2), '3.51 ± 0.1' (Table 1 mean), and '3.64' (Fig 11). These are close but not the same, and there is no stated aggregation rule; a reader cannot tell which is authoritative.
Per-configuration $\alpha$ spread is much wider than the reported '3.5–3.6'. Table 1's bf16 $\alpha$ ranges from 2.86 to 3.94 — a $>1$ bpp span — yet the prose describes the estimate as tightly clustered. Why 1-layer d=256 and 8-layer d=32 sit ~30% below the mean is left unexplored and is arguably the more interesting quantitative story.
Scaling-law validation claim overstated. Section 5.2.2 says predictions are 'generally within 1.5 points of the true F1 score.' Table 2's row-by-row differences reach 3.9 and 9.3 points at the 0.75 target — three of six rows exceed 1.5. Endpoints (0.55 and 0.95) fit well; the middle of the sigmoid does not. This is meaningful and should be reported rather than masked.
Table 2 label error. The 'GPT2-Medium' row lists 768 width, 12 layers, ~124M params — those are GPT-2 small, and the main text says the two validation models are GPT-2 small and GPT-2 XL. Either the row label is wrong or the main text is.
Formula for $H^K(x\mid\theta,\hat\theta)$ conflates a likelihood with a code length. The paper writes ' $\max\{p(x\mid\hat\theta), p(x\mid\theta)\}$ ' as if that quantity is a bit-length. It is not; the code length is $-\log_2$ of the better likelihood. The intent is correct but the display is dimensionally wrong.
Capacity is a lower bound, and its looseness is not quantified. The paper acknowledges that Adam-SGD gives only a lower bound, but never bounds how loose it is. Some of Table 1's 40%-wide $\alpha$ spread across configurations may itself be an optimisation gap rather than a genuine capacity difference — which would sharpen or muddle the headline number depending on which cells sit near optimum.
Extrapolating the F1 law from ~20M targets to production LLMs is a large leap. The fit uses targets up to ~20M and is externally validated only at 1.5B; the paper then projects to 8B–70B / 15T-token regimes. The extrapolation range and residual uncertainty should be stated explicitly.
Reference-shrinkage caveat asserted but not probed. A.13 argues MEM $_U$ starts attributing generalization to memorization as the reference shrinks toward the target, but the sweep never enters that regime (smallest reference ~6× larger than the largest target), so the claim is speculation, not demonstration.
Counterfactual memorization not obviously a special case. The paper claims Zhang et al. 2023's counterfactual memorization 'can be seen as an instantiation' of MEM $_U$ with a family-member reference. But leave-one-out references differ in training data whereas this paper's references differ in scale and corpus breadth; the equivalence should be argued or softened.
Monotone-in-every-cell claim is single-seed. A.13 asserts monotonicity in reference size across 24 cells × 3 references from a single seed (with a second seed only at $d_\mathrm{model}=128$ ). Either add seeds or hedge to 'monotone in expectation, verified in every observed cell'.
'Strictly better estimator' overstates what compression buys. A better compressor tightens an upper bound on $H^K$ ; it is not a 'strictly better estimator' of an uncomputable quantity in any statistical sense. Please say 'strictly tighter upper bound'.
'Grokking' is misattributed to Nakkiran et al. (2019), which is the double-descent paper. Grokking is a distinct phenomenon (Power et al., 2022). Everywhere else Nakkiran is used correctly for double descent.
Kaplan-rebuttal 'flat by construction' overstates the setup. True only for models that have already learned the marginal — the hedge appears once, then is dropped in the next clause.
TF-IDF is stated with the wrong indexing. $\mathrm{tf}(d,D)$ is defined as 'total number of times word $w$ appears in dataset $D$ ' — a collection-level count, not a document-level term-frequency, and $w$ is a free variable. The subsequent interpretation requires the standard $\mathrm{tf}(w,d)$ .
Figure 10 unit error. '23.9 MB' is really 23.9 Mbits (a factor-of-8 slip), inconsistent with Table 1 and with the figure's own bit-labelled y-axis.
Figure 18 caption inverts subject and predicate. 'The monotone decrease in reference size' should be 'the monotone decrease of MEM $_U$ with reference size'.
Definition 3 argument order swaps target and reference. The prose says 'memorization of $x_i$ in $\hat\theta$ ' but the displayed function writes $\mathrm{mem}^K(\theta,\hat x)$ with no index $i$ ; slot order should match Definition 2 and the prose.
'Independent (but earlier)' framing (of Cohen et al., 2024) is internally tensioned — either concurrent or earlier, not both.

Reproducibility & code

Headline aggregate did not reproduce at reduced training scale. The paper's headline $\alpha\approx 3.6$ bpp was executed via a from-manuscript reimplementation whose per-shape parameter counts match the paper to within 0.1% (e.g. 80,352 vs 80,400 for L1/d32; 3,700,224 vs 3,700,000 for L4/d256), confirming architecture faithfulness. The only bf16 cell that reached the memorization plateau (L2/d64, N=1024) gave $\alpha=2.76$ bpp, a 23% shortfall; other saturated small shapes fell in the ~2.6–3.0 range — systematically below rather than around 3.6. The direction (undershoot) is the one the paper's own 'gradient descent yields a lower bound' caveat predicts, and the reproducer was compute-bound (training steps reduced 20×), so this does not disconfirm the paper. But the paper's methods section reports only step count and batch size, not the wall-clock or job-count profile of the full grid; a reader working from the release would find, as this reproducer did, that a shared cluster with a per-user job cap cannot drain the sweep at paper scale without pre-planning.
Optimizer schedule underlying $\alpha$ is not stated. Section 3.2 names Adam, 10 $^6$ steps, and batch 2048, but no LR, warmup, or weight decay for either the synthetic or the text sweeps. Adam has material capacity-relevant hyperparameter sensitivity at this scale, so a reader cannot tell whether Table 1 is optimiser-tuned or optimiser-limited — and cannot say whether the 40% $\alpha$ spread across cells is architectural or an optimisation artifact.
$H^K$ estimator that produced Table 1 is under-described. Capacity is defined as the gap between analytical entropy and the trained model's code length, but Section 2.3 says only 'entropy under $\hat\theta$ '; A.4's ensemble-compression DP is described without switching cost, $(t,k)$ grid, or state transition, and the $O(n\cdot T)$ complexity only holds under strong assumptions a naive $O(n\cdot T^2)$ DP does not satisfy. Which compressor produced Table 1 (and whether the same one is used for text) should be stated with a pointer to release code.
The 'additional deduplication step' on FineWeb is central but never specified. The text says this step is 'extremely important for faithfully measuring extraction rates', yet granularity, hash, cross-split radius, and the exact release script are not given — reproducing Section 4 extraction and MI numbers cleanly depends on it.
Reference-model fine-tuning for the A.13 ablation is thinly specified. The 124M and 355M references are 'fine-tuned in the same way' (one epoch on FineWeb tail), but no LR/batch/verification is given. The reported 2.5× MEM $_U$ shift depends on the 124M reference's loss, so thin fine-tune specification directly propagates into the ablation headline.
Empirical F1 grid behind Figure 5 is not tabulated. The sigmoidal fit $(c_1,c_2,c_3)=(1.34,-0.034,-33.14)$ is applied to a (capacity, dataset size, F1) grid rendered only in the figure. Neither the triples nor the solver initialisation are given, so refitting from the release requires reconstructing them.
Training details for the GPT-2 XL / small validation runs are missing. These six runs are the paper's principal external validation, but Section 5.2.2 states only parameter counts and target F1s — no corpus, token budget, LR schedule, from-scratch-or-continued, or MI threshold rule.
Figure 1 model sizes are not cross-referenced to Table 1 cells. '170K/500K/2.5M/7M' do not map exactly to any Table 1 row (e.g. 170K does not equal L1/D32's 80K or L1/D64's 185K), so a reader locating the runs in code has to infer the configuration.

Recommended Changes

Essential

Reconcile the headline bpp number. State a single aggregation rule (mean across all cells? slope from Fig 11? bf16 vs fp32 split?) and use the same value in the abstract, Section 3.2 prose, Fig 11 caption, and Table 1. Addresses the 'multiple, inconsistent headline bpp numbers' weakness.
Correct the scaling-law accuracy claim. Replace 'generally within 1.5 points' with an honest description of the row-wise errors in Table 2 (e.g. 'endpoints within ~1 pt, mid-sigmoid up to ~9 pts'), or restrict the claim to endpoint targets. Addresses the 'scaling-law validation claim overstated' weakness.
Fix the Table 2 model label. Either rename 'GPT2-Medium' to 'GPT2-Small' (matching the 124M row and Section 5.2.2) or retrain and report actual GPT2-Medium (355M). Addresses the 'Table 2 label error' weakness.
Rewrite the $H^K(x\mid\theta,\hat\theta)$ display as a negative log-likelihood so the dimensions match a code length. Addresses the 'likelihood vs code length' weakness.
Specify the compression / $H^K$ estimator used for the reported numbers, including the arithmetic-coding scheme, the ensemble-compression DP state, switching cost, and $(t,k)$ grid, and point to the release code. Addresses the ' $H^K$ estimator under-described' reproducibility weakness.
Report optimizer hyperparameters (LR, warmup, weight decay) for the synthetic capacity sweep and the text sweep. Addresses the 'optimizer schedule unspecified' reproducibility weakness.
Detail the additional FineWeb deduplication step (granularity, hash, cross-split radius) and link to the exact release script. Addresses the 'additional deduplication' reproducibility weakness.
Report the compute profile needed to reach the paper's plateau (wall-clock per cell, total GPU-hours, whether a paper-scale run is feasible on ≤ 24 concurrent jobs). Addresses the 'headline aggregate did not reproduce at reduced training scale' reproducibility weakness — the reimplementation attempt was compute-bound at 20× fewer steps and could not reach the plateau because the paper does not tell a reader how much compute the plateau costs.

Suggested

Widen the reported bf16 $\alpha$ interval or discuss the architectural dependence — Table 1's 2.86–3.94 span is the interesting story about why $\alpha$ varies across depth/width, and is currently glossed. Addresses the 'per-configuration $\alpha$ spread' weakness.
Add a discussion of how loose the capacity lower bound might be — even one hyperparameter search around a representative cell, longer training for a small cell, or a comparison to an analytic upper bound would help. Addresses the 'capacity is a lower bound' weakness.
State the extrapolation range and residual uncertainty for the 'tokens-per-parameter $\geq 10^2$ → F1 = 0.5' projection. Addresses the 'extrapolating to production LLMs' weakness.
Qualify the reference-shrinkage caveat as 'we expect' or add one data point in the ref≈target regime, so the framework limitation stops reading as demonstrated when it is not. Addresses the 'reference-shrinkage caveat asserted but not probed' weakness.
Add seeds to A.13 or soften the monotone-in-every-cell claim to 'monotone in expectation, verified in every observed cell'. Addresses the 'monotone-in-every-cell single-seed' weakness.
Argue or soften the counterfactual-memorization equivalence — a leave-one-out reference differs from a scale-and-corpus reference in kind, not degree. Addresses the 'counterfactual memorization not obviously a special case' weakness.
Fix 'grokking (Nakkiran et al., 2019)' — either change the citation to Power et al. (2022) or change the phenomenon name to double descent. Addresses the 'grokking misattributed' weakness.
Correct Fig 10's unit from '23.9 MB' to '23.9 Mbits' (and align the inset text). Addresses the 'Figure 10 unit error' weakness.
Rewrite the Fig 18 caption to make MEM $_U$ the subject that decreases. Addresses the 'Figure 18 caption inverts subject' weakness.
Repair the TF-IDF definition as $\mathrm{tf}(w,d)$ counted in document $d$ . Addresses the 'TF-IDF stated with wrong indexing' weakness.
Soften 'strictly better estimator' to 'strictly tighter upper bound'. Addresses the 'strictly better estimator overstates' weakness.
Hedge 'Kaplan's curve is flat by construction' with the 'once the marginal is learned' qualifier. Addresses the 'Kaplan-rebuttal overstates' weakness.
Realign Definition 3's argument order with Definition 2 and the surrounding prose. Addresses the 'Definition 3 argument order' weakness.
Rephrase 'independent (but earlier)' to remove the contradiction with Cohen et al. (2024). Addresses the 'independent but earlier' weakness.
State the exact $(n_\mathrm{layer}, d_\mathrm{model})$ configurations for Fig 1's 170K/500K/2.5M/7M lines in the caption. Addresses the 'Figure 1 model sizes not cross-referenced' reproducibility weakness.
Tabulate the empirical (capacity, $|D|$ , F1) triples fit in Figure 5, or point to a CSV in the release. Addresses the 'empirical F1 grid not tabulated' reproducibility weakness.
Document the GPT-2 XL / small validation-run training details (corpus, token budget, LR, from-scratch-or-continued, MI threshold rule). Addresses the 'training details for validation runs missing' reproducibility weakness.
Specify the reference-model fine-tuning setup (LR, batch, held-out perplexity check) for 124M and 355M. Addresses the 'reference-model fine-tuning thinly specified' reproducibility weakness.
State explicitly that the synthetic capacity pipeline reduces to $H(X) - H_{\hat\theta}(x)$ with no reference model, and that this is what Table 1 computes. Addresses the 'synthetic-sweep reduction never stated explicitly' weakness.