dnaHNet: A Scalable and Hierarchical Foundation Model for Genomic Sequence Learning

Summary

dnaHNet adapts the H-Net dynamic-chunking architecture to prokaryotic genome modelling, arguing that end-to-end learned segmentation resolves the long-standing tension in genomic foundation models between fixed tokenization (which fragments codons, promoters, and other biologically meaningful units) and nucleotide-level modelling (which is computationally prohibitive at genome scale). The conceptual move is to replace a discrete, upstream tokenizer with a differentiable boundary predictor that emits a compressed latent stream for a downstream main network, applied recursively so that the model can capture multi-scale structure — codons in the first stage, functional regions in the second. Empirically, the paper reports that a two-stage (3,2) configuration attains lower perplexity than a compute-matched StripedHyena2 and a Transformer++ across a $4\times10^{18}$–$8\times10^{19}$ FLOPs sweep, wins zero-shot on MaveDB variant-effect prediction and on DEG gene essentiality, and delivers meaningful wall-clock savings at long contexts on an H100. Section 4.6 further presents an interpretability story in which Stage 1 develops triplet periodicity and Stage 2 sharpens on Shine-Dalgarno motifs, −10 boxes, promoters, and intergenic regions. The overall contribution — a first careful scaling study of end-to-end dynamic chunking on genomes — is timely and useful, and the biology-motivated tuning (compression ratios matched to codons and codon-pair bias, smaller auxiliary $\alpha$, redistributed encoder-decoder capacity) is a reasonable set of practical choices. The main conceptual limitation is that the strongest claims — a persistent SOTA advantage, a compounding efficiency margin at scale, and biologically meaningful hierarchical structure — rest on comparisons that are only partially matched (single-seed, no comparison against the actual Evo-2 checkpoint or against the paper's own dynamic-tokenization competitors), on extrapolations well beyond the 1B-parameter ceiling, and on interpretability from five 49-kb windows of a single genome. These are fixable with additional experiments and clearer reporting, but as written they materially shape how much of the paper's headline story can be taken as established.

Strengths

• Conceptual contribution. Framing dynamic chunking as a solution to the tokenizer-vs-nucleotide tradeoff in genomics is well motivated, and applying H-Net at scale to prokaryotic genomes is a natural and non-trivial extension whose scaling behavior had not been previously reported.
• Biology-informed tuning. The design choices in Section 3.2 — target compression $R_1 = 3, R_2 = 2$ aligned with codons and codon-pair bias, a smaller auxiliary loss weight ($\alpha \approx 0.01$), and a heavier encoder-decoder allocation (~30% vs ~15% in text H-Nets) — are principled and explicitly motivated by genomic priors rather than picked post hoc.
• Interpretability signal. Table 1's Stage-1 codon periodicity (position-1 6.5%, position-2 42.6%, position-3 58.4%) and Stage-2 alignment with SD motifs (84.5% exact match), −10 box, promoters, and intergenic regions is a striking qualitative result — the kind of emergent biological structure that would be genuinely useful if it survives broader replication.
• Confound analyses. Appendix A.7 (synonymous-normalized AUROC, pooled z-AUROC, length/GC stratification, per-genome AUROCs in Table 9) is the strongest part of the empirical evaluation — it is the sort of rigor that other genomic-LM papers routinely skip, and it substantially raises the credibility of the DEG results for the (3,2) configuration at higher compute budgets.
• Efficiency measurements at long contexts. The wall-clock, memory, and latency numbers (e.g., 9.7 GB vs 34.3 GB and 410 ms vs 2,352 ms at $2^{18}$ nt; StripedHyena2 OOMing at $2^{19}$) are the kind of concrete, decision-relevant data that engineers will value.

Weaknesses

• The '3.75× more efficient' headline is an extrapolation, not a measurement. The 3.75× compute factor at matched perplexity is derived by extending the fitted PPL power law from the measured range ($8\times10^{19}$ FLOPs) to $3\times10^{20}$ FLOPs. The paper does not report the prefactor $A$, does not report fit uncertainty, and does not disambiguate this training-compute extrapolation from the separately quoted 3.89× inference-FLOP factor at $10^6$ nucleotides or the abstract's ">3× speedup". Combined with the very small fitted exponents ($\alpha = 0.06$ vs $0.04$ vs $0.01$), and with the Transformer baseline being essentially flat with compute — a strong claim in itself — the scaling narrative sits on a fragile numerical foundation.
• State-of-the-art claim is broader than the empirical comparison. The Abstract, Introduction, and Section 3 introduce dnaHNet as a "new state-of-the-art for genomic foundation models", but the empirical comparisons are limited to compute-matched retrains of StripedHyena2 and Transformer++. There is no comparison against a released Evo-2 checkpoint on the same MaveDB / DEG tasks, and no comparison against the dynamic-tokenization competitors the Related Work section flags as directly relevant — MergeDNA, PatchDNA, MxDNA. Without those, "SOTA" reads as "beats our compute-matched StripedHyena2 retrain", which is a defensible but much narrower claim.
• Interpretability rests on 5 windows from a single genome. The Section 4.6 numbers (Table 1) are computed on five random 49,152-bp windows of a single $B.\ subtilis$ assembly, with no error bars, no permutation/GC-matched control, and no cross-genome (let alone held-out-taxonomy) replication. The specific percentages are compelling, but their generalization to bacteria (let alone to the eukaryotic future work the Discussion invokes) is not established.
• Missing / inconsistent equations and notation. The Decoder smoothing recurrence is introduced by a colon that has no equation attached; the ratio-loss functional form combining $F_s, G_s, R_s$ is likewise missing. Together with the underspecified inference-time "accumulation" (Section 3.3) that does not clearly match the training-time boundary selection (Section 3.1), the architecture as written is not self-contained — a reader must fall back on Hwang et al. for load-bearing components.
• Overclaimed scope of the FLOP reductions. The 36× main-network FLOP figure is really $R^2$ savings on the attention-score term; per-token linear costs (QKV / output projections, MLP) only drop as $R$, and at an inner sequence length of ~1365 tokens at $D \approx 800$ the linear terms often dominate. Similarly, the Abstract's "quadratic FLOP reductions" drops the "for the main network" qualifier that Section 4.3 correctly attaches, allowing readers to infer end-to-end quadratic savings that the architecture does not deliver.
• Confound conclusion overgeneralizes across compute scales. "Performance remains well above chance under all three controls" is defensible for the (3,2) configuration at higher compute, but at $8\times10^{18}$ FLOPs the (2,2) synonymous-normalized macro AUROC is $0.512 \pm 0.017$ (Table 7) — statistically indistinguishable from chance. The blanket phrasing hides where the signal fails.
• Metric aggregation mismatch. Table 6's headline AUROCs and Table 7's macro-raw AUROCs differ (e.g., (3,2) at $6.4\times10^{19}$: 0.6050 vs 0.599; the tables also diverge qualitatively at the top budget, with Table 6 continuing to climb while Table 7 plateaus). The two aggregations are never labelled as such, and the reader cannot tell which is the "official" DEG number.
• Cross-reference and variant-labelling errors. Figure 6 is cited for both the codon/functional-region analysis and the wall-clock sweep; Section A.7 points to "Section 4.3" for DEG results that live in Section 4.5; Figure 2 compares dnaHNet-218M and StripedHyena2-166M, while Section 4.7 uses M/L/XL vs SH2-100/167/234M without stating which is which. These small errors compound because they touch different results.
• Contradictory description of the DEG knockout perturbation. The main text says variants are generated by "inserting a 15-bp stop codon sequence 12 bp downstream of the start codon (15 nucleotides were replaced)". Insertion and replacement have different length semantics that directly affect the WT-vs-variant log-likelihood comparison.
• GC-tertile row for (3,2) coincidentally equals (2,2) overall. The (3,2) length-tertile means average to the (3,2) overall AUROC as expected, but the (3,2) GC-tertile means average to 0.585 — exactly the reported (2,2) overall. Given how tightly the length tertiles match the overall for (3,2), the coincidence looks like a possible transcription error.
• Single-seed reporting. No variance estimates are reported for perplexity, Spearman correlation, or AUROC. Several headline gaps are small (perplexity gap of 0.078 at $8\times10^{18}$ FLOPs; (2,2) vs (3,2) DEG differences of 0.01–0.02), and would benefit from being placed against seed variance.

Reproducibility & code

Veritas reviewed the paper against the released resources; no code or trained checkpoints were found for dnaHNet and the reproducibility risk on every headline claim is rated medium-to-high without them.

• No code-availability statement or checkpoint release. The manuscript has no repository link and no data/code-availability section. All headline results (VEP correlations, DEG AUROCs, chunking-selection percentages, wall-clock/memory numbers) require trained checkpoints; without them, and given that several load-bearing equations are missing (smoothing recurrence, ratio-loss form), a reader would have to re-implement dnaHNet from prose. This is the single largest reproducibility risk.
• Scaling-law sweep grid not tabulated. The reported exponents ($\alpha$) and the 140B/68B compute-optimal points depend on a (params, tokens, batch, PPL) sweep of ~100 models that is not released, and on a "compute-optimal frontier" whose selection rule is not stated. StripedHyena2 and Transformer++ training recipes (learning rates, tokenizer, data-mixing) are also not documented — the Transformer's near-flat $\alpha \approx 0.01$ is impossible to attribute (architecture vs tuning) without them.
• Evaluation-set specifications are underspecified. The 12 MaveDB E. coli datasets are not enumerated by URN; the B. subtilis assembly accession, five-window RNG seed, and motif-matching tolerance for Table 1 are not given; the synonymous-codon substitution rule and window-centering procedure for DEG are not spelled out; and Appendix A.5's tokenized Transformer++ baselines do not state BPE vocab size or the FLOP-matching formula used.
• Wall-clock benchmark configuration underspecified. Section 4.7 documents H100/BF16 and mentions Triton, TorchInductor, and DeepSpeed Stage 2, but not batch size, warmup, number of forward passes averaged, kernel/library versions, or whether trained or randomly-initialized weights were benchmarked — all of which materially affect throughput and memory. The qualitative outcome (dnaHNet-XL faster and smaller than SH2-234M) is plausible from the architecture; the specific 6.0× / 3.5× / 5.7× ratios are much harder to reproduce.
• Missing equations reduce self-containment. The decoder smoothing recurrence and the ratio-loss functional form appear to be introduced by display anchors with no equation rendered. Both are load-bearing training/inference components; without them the paper is not reproducible from equations alone, and readers must fall back on the H-Net reference.
• Encoder/decoder hybrid ordering. The 4 Mamba + 1 Transformer composition is stated but the interleaving order and sliding-window size are left unspecified. Layer ordering in hybrid stacks is known to matter and should be documented (or the H-Net convention explicitly inherited).

Recommended Changes

Essential

• Release code and checkpoints, and add a data/code-availability statement. Publish a repository containing the model implementation, training scripts, evaluation harnesses, and at least one trained checkpoint (dnaHNet-XL, (3,2)), with library / kernel versions used for the wall-clock benchmarks. Ties directly to the Reproducibility & code weakness about no released code and to every reproducibility comment on evaluation specifications.
• Provide the scaling-law sweep table and prefactors, and disambiguate the '3×-ish' factors. Release a per-run table (params, tokens, batch, seed, val PPL, FLOPs) for all ~100 models in Section 4.3 and report the fitted prefactor $A$ and uncertainty for each architecture family. In the discussion, separate the 3.75× training-compute extrapolation, the 3.89× inference-FLOP figure, and the >3× wall-clock speedup, and flag the 3.75× as an out-of-sample projection. Addresses the perplexity-exponent weakness and the 3.75×/3.89× ambiguity.
• Add missing methods content: smoothing recurrence, ratio-loss form, encoder ordering, streaming inference. Render the display equations that Section 3 introduces but does not typeset, specify the 4-Mamba + 1-Transformer interleaving and sliding-window size, and rewrite the Section 3.3 inference paragraph so that 'accumulation' is either identified with training-time boundary selection or defined precisely. Addresses the missing-equations and inference-consistency weaknesses.
• Clarify insertion vs substitution in the DEG knockout, and add per-URN specification for MaveDB. Fix the "inserting … (15 nucleotides were replaced)" contradiction in Section 4.5; state which of the two operations was done and rerun the analysis if needed. Enumerate the 12 MaveDB E. coli URNs and per-study point counts. Ties to the DEG-knockout weakness and the MaveDB reproducibility comment.
• Reconcile Table 6 (headline DEG AUROC) with Table 7 (macro-raw AUROC). Label each with its aggregation (macro vs pooled/micro) and either reconcile the top-budget discrepancy (0.6050 vs 0.599) or explain why the headline metric continues to rise where macro-raw plateaus. Addresses the metric-aggregation weakness.
• Rerun the smallest headline configurations with multiple seeds and report variance. At least for the (3,2) headline points on VEP and DEG, and for the perplexity gap at $8\times10^{18}$ FLOPs, report mean ± SD across seeds (or bootstrap CIs on Spearman/AUROC). Addresses the single-seed weakness.

Suggested

• Compare against Evo-2 and against the cited dynamic-tokenization competitors. Evaluate a released Evo-2 checkpoint and the MergeDNA / PatchDNA / MxDNA baselines zero-shot on MaveDB and DEG, or narrow the "state-of-the-art" claim to compute-matched comparisons against StripedHyena2 and Transformer++. Ties to the SOTA-scope weakness.
• Strengthen the interpretability analysis. Expand Section 4.6 to multiple genomes (including at least one held out from training), report window-level dispersion for the Table 1 percentages, add a permuted / GC-matched control, and pin down the assembly accession, five-window seed, and motif-matching tolerance so the analysis is reproducible. Ties to the interpretability weakness and the B. subtilis reproducibility comment.
• Qualify the abstract's efficiency claims and the "grows with scale" narrative. In the Abstract, change "quadratic FLOP reductions" to "quadratic reductions in main-network attention FLOPs" (or equivalent). In the Discussion, replace "a substantial efficiency margin that grows with model scale" with a wording that explicitly marks any trend beyond 1B as projected. Addresses the FLOP-scope and 1B-ceiling weaknesses.
• Scope the "well above chance" statement in Appendix A.7. Add a per-row significance test vs 0.5 or restrict the sentence to the (3,2) configuration at the higher compute budgets. Ties to the confound-generalization weakness.
• Fix cross-references and variant labelling. Give the biology figure a distinct number from the wall-clock figure; correct the "Section 4.3" reference in A.7 to "Section 4.5"; state which of M/L/XL corresponds to 218M and confirm SH2-166M / SH2-167M are the same model. Ties to the cross-reference weakness.
• Double-check the (3,2) GC-tertile row in Table 8. Verify whether the GC-tertile entries were computed from the (3,2) model rather than accidentally reused from (2,2); rerun if needed. Ties to the GC-tertile weakness.
• Rewrite the contribution bullet on "compression ratio scheduling" and "initialization strategies". Either describe a schedule and an initialization protocol in Section 3.2, or soften the contribution bullet to only include the methods actually presented. Ties to the contribution-list weakness.
• Separate FLOP-accounting effect from information-efficiency claim. Rewrite the "extract more signal per raw nucleotide" sentence so the compute-allocation mechanism (fewer FLOPs per token → more tokens preferred) is distinguished from an information-efficiency mechanism. Ties to the mechanistic-story weakness.
• Tighten the Table 3 α range. Either narrow the reported [0.01, 0.05] auxiliary-loss weight range to values actually used in reported runs (0.02–0.03), or state where 0.05 was exercised.