TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Summary

TokSuite addresses a genuinely under-studied confound in modern LM research: because production tokenizers vary along many entangled axes (vocabulary size, learning algorithm, pretokenization, normalization, training corpus), attributing any behavior of a released model to its tokenizer specifically is fraught. The authors' conceptual move is to fix everything except the tokenizer — 14 1B-parameter Llama-style models are trained on the same multilingual corpus, from a shared initialization built on a novel "super vocabulary" that unifies token IDs across tokenizers, using an identical 100B-token budget. They then evaluate all 14 models on a purpose-built benchmark of ~5,000 items across English, Farsi, Italian, Turkish, and Chinese, together with math and STEM subsets, that isolates real-world perturbations known to change tokenization but not meaning (romanization, homoglyphs, zero-width characters, morphological variation, Unicode styling, LaTeX). The core empirical finding — that a small 32K English-only tokenizer (TokenMonster) can top a multilingual robustness ranking while a 259-token byte-level tokenizer (ByT5) is competitive — is genuinely interesting, and the accompanying scaling, text-matched, seed, and industry-model ablations show admirable methodological care; the intrinsic-metric tables (fertility/parity/PCW) and the scale-invariance result cleanly reproduce cell-for-cell in independent execution, and the TokenMonster headline reproduces at 0.183 ≈ 0.17. The main conceptual limitation is that studying off-the-shelf tokenizers necessarily leaves many design axes co-varying, so mechanistic causal claims (e.g. TokenMonster wins "because of its ungreedy algorithm and consistency constraints") outrun the identification strategy the design supports. Two methodological choices — a fixed token budget (which lets ByT5 see 100 GB while Gemma-2 sees 471 GB) and a byte-length-normalized log-likelihood scorer — are entangled with exactly the quantity being measured, and the relative-drop robustness metric can advantage models with lower canonical baselines. An independent execution also shows that several TokSuite base checkpoints (mBERT, XGLM, ByT5) evaluate at near-chance canonical accuracy, and the Table 1 "Avg" row only recovers the paper's reported category aggregates once these degenerate rows are effectively excluded; this filtering is not documented. Taken together the paper is a valuable, careful, and well-executed contribution to a genuinely neglected area; the identification and metric choices deserve more upfront discussion, several mechanistic claims should be softened to match what the design actually licenses, and the aggregation rule used to produce Table 1's row/column averages needs to be spelled out.

Strengths

• Conceptual contribution. Isolating tokenizer effects via a shared super-vocabulary initialization and matched architecture/data/budget is a substantive step beyond prior "train tokenizers from scratch" designs (Ali et al. 2024; Schmidt et al. 2024), and enables comparisons practitioners actually face when selecting from off-the-shelf tokenizers.
• Benchmark design. Native-annotator curation of realistic perturbations across five typologically diverse languages (Farsi diacritics, Pinyin/Finglish romanization, Turkish agglutination, Italian elision) fills a real gap over adversarial-attack or synthetic-noise robustness datasets, and the "canonical questions almost all models get right" protocol cleanly measures degradation rather than baseline capability.
• Statistical hygiene. The paper reports 10,000-trial bootstrap CIs, Wilcoxon signed-rank tests for pairwise significance, and a seed-sensitivity ablation (Appendix G.3) — a level of care not standard in tokenizer papers.
• Confound-management ablations. Text-matched training (Appendix G.2), model scale (Appendix G.1), Table 5's raw-byte accounting, and the industry-model comparison (Appendix G.4) each directly address obvious confounds and are appropriately scoped.
• Concrete case studies. Section E's per-tokenizer segmentation traces (Turkish "saatteki" vs "saatdeki", Italian "dell'Italia" vs "d'Italia", scientific notation with braces) turn abstract robustness numbers into legible mechanistic vignettes.
• Intrinsic-metric tables reproduce cleanly. Independent re-execution recovers every one of the 70 fertility cells and 56 parity cells within ±0.05 of Table 4, and PCW within ±0.05 across 70 cells — the intrinsic-tokenizer story is a solid, auditable base for the rest of the paper.
• Open release. Releasing 14 checkpoints, the perturbation code, the detokenized training data, and lm-eval-harness integration is exactly what future tokenizer research needs and will likely be picked up as an infrastructure contribution.

Weaknesses

• Fixed token-budget confound. With ByT5 consuming 100 GB and Gemma-2 consuming 471 GB (Table 5), models are not "otherwise equivalent" — they see nearly 5× different amounts of underlying information. The Section 3.3 sentence "this performance difference can be attributed directly to tokenization" is stronger than the design licenses; the text-matched control (Appendix G.2) covers only 4 of 14 tokenizers, with per-model deltas of ±0.02, which is helpful but does not close the confound at scale.
• Relative-drop metric can favour weaker baselines. $(\text{Acc}_{\text{can}} - \text{Acc}_{\text{pert}})/\text{Acc}_{\text{can}}$ has less room to move when canonical accuracy is already low. TokenMonster's canonical Avg (0.76, IT=0.68) is the lowest in Table 6, and its top position in the robustness ranking may partly reflect the metric's denominator asymmetry. Reporting absolute drops alongside, or restricting to items every model gets canonically right, would sharpen the interpretation.
• Overreaching mechanistic attributions. "This effectiveness stems … from its unique 'ungreedy' tokenization algorithm and consistency constraints" attributes TokenMonster's win to a single design axis, but consistency is never measured and TokenMonster differs from other tokenizers along many axes at once. The same over-attribution pattern appears for BLOOM's diacritics win ("due to its multilingual design") and XGLM's Unicode robustness — the multilingual claim in particular is directly contradicted by other multilingual tokenizers that fare worse.
• Scale ablation is single-tokenizer. Table 22's "minimal variation" across 300M/1B/7B uses only the Llama-3.2 tokenizer, so the generalization "tokenizer design rather than parameter count dominates robustness" is not identified. The 7B model also drops 0.04 (~15% relative) below the 1B value — reproduced in the veritas run as 0.26/0.23/0.21 — which the "minimal variation" framing understates.
• Scoring rule is entangled with tokenization. Robustness is computed using lm-eval's byte-length-normalized log-likelihood. Byte-length normalization is not neutral across tokenizers, since it interacts with how token boundaries in candidate answers align with byte counts. A robustness check against token-length or character-length normalization would rule out that part of the "tokenizer effect" lives in the scorer.
• Base models are absolutely weak. By the authors' own admission (Appendix C.2), performance on Belebele/INCLUDE "hardly exceeds the random baseline." A robustness ranking measured in this low-capability regime may not preserve at industry scale; Table 28 already shows a compressed spread and a different top model (XGLM-564M, not TokenMonster).
• Small benchmark item counts. 40 canonical items per language, 20 math and 44 STEM canonical items mean single-question flips can move category scores by 2–5 points, which is the order of some of the headline gaps in Section 5.
• Confounded off-the-shelf axes. Off-the-shelf tokenizers differ along vocab size, algorithm, normalization, pretokenization, and training data simultaneously. Figure 12's "grouping by algorithm" glosses over this — most notably by treating Gemma-2 as a single algorithm class despite Table 2's honest note that paper and HuggingFace implementations disagree (Unigram vs BPE).
• Notation and metric-definition ambiguities. The super-vocabulary framework calls its map "bijective" when it is at most injective, and the symbol $SV$ is overloaded as (a) the union set, (b) a mapping applied to strings, and (c) a mapping applied to integer IDs. The parity metric's "$\approx 1$ when fair" definition contradicts Table 4/Figure 4's "lower is better" caption. The GPT-4 digit-grouping example ("1337" → "133","7") is left-aligned, contrary to actual cl100k behavior.
• Case-study inconsistencies. The ASCII-methane paragraph describes BLOOM/GPT-4o/Tekken as "preserves structure well" yet then names only Llama and GPT-2 as correct. The LaTeX braced-cubic-meters case names TokenMonster and Qwen-3 as robust, but Qwen-3's enumerated tokenization is as fragmented as the failing models. Specific numeric claims — "6/14 broken by week → weak"; ByT5's "-0.11" zero-width improvement — cannot be traced to the referenced tables.
• Weakly supported negative result. "We don't observe significant correlations between intrinsic metrics and the downstream performance" is one of the paper's headline contributions and is asserted in a single sentence, without a correlation coefficient, plot, or p-value. Given the paper's careful bootstrap+Wilcoxon apparatus elsewhere, this deserves a proper analysis.

Reproducibility & code

• TokenMonster headline reproduces; ByT5 headline does not. Rerunning the shipped get_table_str pipeline places TokenMonster's average drop at 0.183 (within ±0.03 of the reported 0.17), confirming the paper's central finding at scalar level. In the same run, however, ByT5's average drop reproduces as 0.09 rather than 0.22 — qualitatively even more robust than reported, but the specific anchor does not match.
• Table 1 aggregates require an undocumented filter. Rerunning end-to-end reproduces 129 of 180 Table 1 cells within ±0.05, but three reproduced TokSuite base models (mBERT, XGLM, ByT5) evaluate at near-chance canonical accuracy (0.34–0.54) and generate strongly-negative "drops" (e.g. mBERT STEM −0.77, XGLM STEM −0.65). Including these in the Avg row gives 0.18 rather than the paper's 0.24; excluding them recovers the reported LaTeX ≈ 0.22 and STEM ≈ 0.26. The paper never says the Avg row filters out degenerate baselines, and the reported 0.28 STEM average is arithmetically incompatible with including such rows. The filtering rule (or checkpoint version) that produced the reported aggregates needs to be documented.
• Intrinsic-metric tables reproduce cleanly. Fertility (70 cells), parity (56 cells), and PCW (70 cells) all match Table 4 within ±0.05 after re-executing calculate_intrinsic_tokenizer_metrics.py. This is the most solid piece of the reproducibility picture.
• Scale ablation reproduces qualitatively. Re-training/re-evaluating Llama-3.2 at 300M/1B/7B gives (0.26, 0.23, 0.21) versus the paper's (0.26, 0.26, 0.22) — all three points within the ±0.03 target, and the reproduced 7B step is slightly larger than the paper's, which strengthens (not weakens) the caveat about the "minimal variation" framing.
• LaTeX braces case study only partially recoverable. For the volume question (m^3), the braced 2/14 collapse reproduces exactly, but the reproduced STEM benchmark has no distinct "spaced-braces" variant of the question, so the paper's 2/14 spaced-braces anchor cannot be checked from the shipped item set. Plain-text and basic-LaTeX counts (10/14 and 10/14) also disagree with the paper's (14/14, 8/14) at 1B scale.
• Bootstrap seed and Wilcoxon script. No explicit random seed is documented for the 10,000-trial bootstrap and no standalone bootstrap CLI is obvious in the shipped repo (the computation appears embedded in all_tables.ipynb). Because Section 5 comparisons sit close to the ±1 SD threshold, the CI values are not bit-for-bit reproducible.
• Text-matched corpus not shipped. Appendix G.2 is the paper's main answer to the token-budget confound, but the reconstructed rank-specific corpora used to train Llama-3.2/Gemma-2/Comma/Qwen-3 on identical bytes are not obviously released, and no reconstruction script was found. Third parties would have to log a full Llama-3.2-1B training run themselves.
• 300M and 7B scaling checkpoints. The standard HF release under toksuite/ lists one 1B checkpoint per tokenizer; the 300M and 7B Llama-3.2 variants that ground the scale-invariance claim would need to be retrained from the Lingua toksuite branch configs.
• EBL modifications live only in prose. Appendix F relies on "extensive manual modification" to Phan et al. (2025)'s Exact-Byte-Level code to handle Chinese and Farsi. Those modifications are not shipped, and because they change the delimiter rule that gives EBL its statistical guarantee, both correctness and reproducibility of Tables 20–21 depend on them.
• BPB training logs and byte-counting script. Figure 6's convergence claim and Table 5's raw-byte counts are not backed by shipped logs or a byte-counting script; a small CSV per model plus a one-liner would make these numbers auditable at essentially no cost.
• Repo requires substantial re-engineering to run. Executing the pipeline outside the authors' SLURM cluster required replacing the shipped launchers, rewriting all_tables.ipynb's hardcoded cluster paths, hand-selecting per-model tokenizer backends (tokenmonster/tekken/tiktoken), pre-generating the GPT-4o tiktoken vocab, adding OOM/GPU-retry logic, and swapping two gated repos (Llama-3.2, Aya) for byte-identical ungated mirrors. A dead crashing import in calculate_intrinsic_tokenizer_metrics.py (from toksuite.utils import system) also indicates the intrinsic-metrics script was not re-executed cleanly before release.

Recommended Changes

Essential

• Document the Table 1 aggregation rule. State explicitly which models are included in the "Avg" row and column, and either re-report those aggregates over all 14 models with a caveat, or state the exclusion rule (near-chance canonical accuracy) that produced the reported category means. Without this, the LaTeX 0.22 / STEM 0.28 / overall 0.24 headline aggregates are not recoverable from the shipped code.
• Soften "attributed directly to tokenization." In Section 3.3, revise the sentence following the sanity-check paragraph to acknowledge the fixed-token-budget regime and the 100–471 GB byte-volume spread reported in Table 5.
• Report absolute-drop and canonical-consistent robustness alongside the relative-drop metric. Add either (i) absolute drops or (ii) a "restricted to items all 14 canonical-correct" variant to Table 1, so the TokenMonster/ByT5 ordering can be separated from the denominator asymmetry.
• Qualify or measure the "ungreedy algorithm" claim. For the TokenMonster finding in Section 5, either (a) report a quantitative consistency measurement (variance of token sequences across surface-form variants of the same word) or (b) soften "This effectiveness stems from" to "is consistent with."
• Soften the scale-invariance claim or add a second-tokenizer scaling curve. In Section 5, restrict "tokenizer design ... dominates" to the Llama-3.2 tokenizer, acknowledge the ~15% relative improvement at 7B, or add at least one non-BPE tokenizer at 7B.
• Add a scoring-rule robustness check. In Section 4.3 or the Appendix, report the Table 1 headline differences under an alternative normalization (token-length or character-length) to rule out that byte-length normalization is entangled with tokenizer identity.
• Pin the bootstrap seed and release the CI/Wilcoxon scripts. Add a seed to Section 4.3 "Statistical Validation" and ship a callable script (not only a notebook) so 10,000-trial CIs and Wilcoxon tests are reproducible bit-for-bit.
• Release (or provide a reconstruction script for) the text-matched corpora. Without the reconstructed rank-specific corpora, third parties cannot rerun the text-budget ablation that grounds the paper's defence against the byte-volume confound.
• Ship the modified EBL implementation for non-space-delimited languages. Tables 20–21 depend on modifications that weaken EBL's delimiter assumption; please release the code so reviewers can audit whether the "statistically equivalent byte-level model" property is preserved for Chinese and Farsi.
• Reconcile the LaTeX braces case study with the shipped benchmark. Either add the "spaced-braces" question variant to the released STEM benchmark, or update the "14/8/2/2" text to reflect the item set that actually ships; also reconcile the plain and basic-LaTeX counts against the 1B model set.

Suggested

• Fix the notation issues in Section 3.2. Replace "bijective" with "consistent shared indexing," and disambiguate the overloaded $SV$ with distinct symbols for the set, the map on strings, and the map on IDs.
• Reconcile parity's "lower is better." Update Section 4.3 and Table 4/Figure 4 captions to name the reference language explicitly and state whether the metric is one-sided or two-sided.
• Correct the digit-grouping example. Update Section 2 and Appendix A so that "1337" → "1","337" reflects cl100k's right-aligned grouping; update the "username12345" example in the Group-by-3 paragraph accordingly.
• Name the language-specific word segmenter per language (especially Chinese) in Appendix B so the intrinsic metrics are auditable from the specification alone.
• Attribute Gemma-2 explicitly in per-algorithm groupings. In Figure 12 and any per-algorithm aggregate, either exclude Gemma-2 or annotate which classification (Unigram or BPE) is used.
• Support the "no significant correlation" claim. Add correlation coefficients (Spearman/Pearson) and p-values, or a small scatter plot, between each intrinsic metric and category robustness.
• Reconcile case-study contradictions. Rewrite the ASCII-methane paragraph so that "only Llama and GPT-2 correct" is consistent with the tokenizations enumerated above it, and add a discriminator that separates Qwen-3's fragmented-yet-correct behavior from XGLM/Llama-3.2 in the LaTeX braced case.
• Soften "comparable to vastly different training regimes." Either present the Table 1 and Table 28 spreads explicitly side-by-side or weaken the conclusion's claim.
• Rework "robust to subword fragmentation." In the conclusion, reword this phrase for byte-level models, which do not perform subword tokenization at all.
• Clarify Table 7 counts. Explicitly state that Math=189 and STEM=614 are canonical + perturbation totals and give the multipliers relative to 20 and 44 canonical items.
• Fix or verify small numeric citations. Verify or fix the ByT5 "-0.11 zero-width" citation, and either enumerate the 6/14 model list for the week → weak claim or drop the count.
• Release the 300M and 7B TokSuite-Llama checkpoints, or point at the exact Lingua configs / training commands so the scaling ablation is a one-command reproduction.
• Ship BPB training loss traces and a byte-counting script so Figure 6's convergence claim and Table 5's byte totals can be audited without retraining.
• Rewrite the green/red significance legend ("more than 1 SD below/above the mean") to remove the ambiguous ">1 SD below" phrasing that keys four figures.
• Use instruction-tuned identifiers when comparing instruct vs base. In Appendix G.4, refer to gemma-2-2b-it and Llama-3.2-1B-Instruct rather than their base names.
• Provide a non-cluster reproduction path. Ship a single-node runner and remove the hardcoded cluster paths from all_tables.ipynb, and fix the dead from toksuite.utils import system import in calculate_intrinsic_tokenizer_metrics.py.
• Lower the canonical-accuracy floor claim. Change "consistently exceeds 70–75%" in Section 4 to reflect the several Table 6 cells at or below 0.68.