Protein Autoregressive Modeling via Multiscale Structure Generation

Summary

PAR proposes the first multi-scale autoregressive framework for protein backbone generation, adapting the "next-scale prediction" idea from VAR-style image models to $C_\alpha$ coordinates. Instead of factoring a protein into residue-level tokens — which forces a unidirectional order over strongly bidirectional biophysical dependencies — or discretizing structure with a VQ-tokenizer, which loses fine-grained detail, PAR downsamples a backbone into a coarse-to-fine hierarchy of centroids and predicts each scale from the previous ones using an AR transformer that conditions a shared flow-matching $C_\alpha$ decoder. The framework naturally accommodates two exposure-bias mitigations — noisy context learning and scheduled sampling — and reduces to a single-scale flow-matching model when only one scale is used, so the paper is positioned as a strict generalization of Proteina.

The conceptual move is appealing: making the AR "token" a whole coarse structure sidesteps both discretization loss and the token-order ambiguity that has stymied prior structure-AR work, and the resulting multi-scale interface enables three genuinely useful capabilities — coarse-point prompted generation, zero-shot motif scaffolding by teacher-forcing motif coordinates at every scale, and orchestrated SDE/ODE sampling that reuses cheap coarse steps to amortize expensive fine-scale ODE integration. The scaling study, the exposure-bias ablation, and the attention analysis all provide useful qualitative support for the design.

Overall, PAR is a substantive contribution — a well-motivated architecture that opens a new AR design axis for protein structure. The main conceptual limitation is that most of the paper's headline distributional advantage rests on PDB-specific fine-tuning: the 400M base PAR is not consistently better than Proteina on FPSD, and the abstract / contribution list blur this qualifier. Several ablation, notation, and reproducibility details also need tightening before the empirical picture is as crisp as the framework itself.

Strengths

• Conceptual contribution. Treating downsampled coarse structures as the AR "token" is a genuinely new factorization for protein backbones. It respects the paper's stated design constraints (avoid discretization, avoid a residue-level unidirectional order) and gives a principled bridge between AR and flow-matching by reducing to Proteina at $n{=}1$.
• Zero-shot capabilities. The point-prompted generation (Fig. 3, Table 5) and the teacher-forcing motif-scaffolding recipe (Fig. 4, Table 11) are direct consequences of the multi-scale factorization and would be hard to obtain from a diffusion baseline without extra training heads. Even where absolute success rates are modest, the fact that these tasks work at all with no fine-tuning is a real capability differentiator.
• Sampling efficiency. The SDE-at-coarse / ODE-at-fine orchestration in Table 2 is a clean architectural payoff: an SDE budget spent on the first (short, cheap) scale is enough to place samples in a high-density region, so later scales can integrate with 2 ODE steps and still land at ~97% designability. The 1.96×–2.5× speed-up over the matched Proteina baseline is credible.
• Exposure-bias mitigation. Noisy context learning and scheduled sampling are well-motivated adaptations of language-model techniques to the multi-scale continuous setting, and Table 3 shows a large sc-RMSD improvement (2.20 → 1.48) at the 60M / 100K-step scale.
• Diligent scaling and ablation. The size × training-step sweep, the by-length vs by-ratio scale ablation, the transformer-encoder vs direct-input comparison (Table 5), the self-conditioning ablation (Fig. 8), and the AR / decoder size ablation (Table 13) collectively provide reasonable coverage of the main design choices.
• Attention analysis. The per-scale attention averaging (Figs. 6, 10) gives at least a qualitative mechanistic picture of how the AR transformer routes information across scales, and the observation that the model attends primarily to the previous scale is consistent with the "coarse-to-fine" narrative.

Weaknesses

• Headline FPSD 161.0 is not like-for-like. The 161.0 FPSD-vs-PDB is achieved by PARpdb, which is fine-tuned on the paper's re-derived 21K PDB subset — a subset that is not given to the baselines. Since FPSD-vs-PDB is exactly the metric on which fine-tuning on PDB should help, quoting 161.0 in the contribution list without the qualifier overstates the like-for-like advantage. Base PAR (400M) is at 313.9, worse than Proteina (271.3).
• "Competitive on FPSD" is only true under two specific configurations. In the like-for-like regime (no PDB fine-tuning, default $\gamma$), 400M PAR is behind Proteina on FPSD vs both PDB (313.9 vs 271.3) and AFDB (296.4 vs 272.6). Only $\gamma=0.45$ (231.5 / 211.8) or PDB fine-tuning improves this. The intro / discussion framing does not make that condition-dependence visible.
• Table 3 caption "consistently effective" is contradicted by the same table's FPSD (89.70 → 90.66 when adding SS) and by mildly worsened fS-A. sc-RMSD is monotone; the distributional metrics are not. The caption also contains "effective effective" as a repeated word.
• Scaling FPSD 187 / 170 is not reconcilable with Table 1 for any comparable 400M PAR configuration reported elsewhere, and Fig. 5's underlying values are not tabulated. Without a table or a clear statement of the sampling configuration and step count, these numbers cannot be verified against the rest of the paper.
• Sampling-step configuration is internally inconsistent. §A.3 says PAR uses 1k steps while Table 8 lists PAR steps as 400/2/2 and Table 2 explores 400/400/400, 400/400/2, etc., with no 1k configuration. A reader cannot reproduce Table 1 without knowing which schedule was used.
• 4.7× vs 2.5× speed-up are measured against different baselines (PAR-internal all-SDE vs Proteina 400-step) but presented adjacently without label. Readers can easily conflate them.
• Scale-count non-monotonicity contradicts the "error accumulation" attribution. Table 4 shows the 4-scale designability at 77.8% and the 5-scale at 81.0%; under a simple error-accumulation-with-more-steps mechanism the ordering should be the opposite.
• AR-size ablation understated. The paper says increasing AR from 60M to 400M "does not offer improvements", but Table 13 shows it degrades both sc-RMSD (1.19 → 1.26) and designability (92.6% → 87.8%). The exposure-bias framing that follows is only coherent with a regression.
• Table 14 leans on an uninformative LDDT. LDDT saturates at exactly 1 across all downsampling sizes, including sizes where RMSE is 0.362; the metric therefore carries no discriminative signal for the claim that spatial relationships are preserved.
• Bicubic interpolation of a distance map is not a valid geometric reference. The "ground truth" downsampled distance map in §C.8 is itself an interpolation artifact — it is not the distance matrix of any subsampled set of 3D points — so the RMSE / LDDT comparison is between two approximations, not against a canonical reference.
• Reduction to flow-based model is not quite to a vanilla flow model. At $n{=}1$ the decoder is still conditioned on $\mathbf{z}^{1}$ produced from a learnable bos embedding, so PAR reduces to a conditional flow, not the unconditional / Proteina case — a small but load-bearing qualifier when the paper claims strict generalization.
• Motif scaffolding framing. Emphasizing that PAR is "the only one evaluated in a zero-shot setting" is fair, but Table 11 shows fine-tuned baselines dominate on most cells (e.g., 71.3, 41.5, 40.3 vs PAR* fractions of a percent). The wording invites reading the table as competitive-outperformance when it is really a capability trade-off.
• No residue-level AR control. The empirical comparisons are entirely against diffusion / flow baselines or against a token-language model. To attribute PAR's zero-shot generality specifically to the multi-scale factorization (rather than to the flow decoder + Proteina backbone), a matched residue-level continuous-AR baseline — or at minimum a discussion of why one is not run — is needed.
• No seed variance or CIs anywhere. Table 1 gaps of 0.08 Å in sc-RMSD, the small FPSD deltas in Tables 4 / 12, and 100-sample designability estimates all sit within plausible Monte-Carlo noise; without seeds or bootstrap intervals, several ablation conclusions rest on point estimates.
• Abstract/contribution mismatch on zero-shot generalization. "Strong zero-shot generalization" (abstract) vs "zero-shot generalization potential" (contribution vi) sends two different signals about the strength of the demonstrated result.
• Auxiliary claims presented as fact. The VAE-latent "smoother Gaussian space that is robust to small errors" analogy is used as load-bearing motivation for keeping the AR small but is technically imprecise; the assertion that "scaling the AR transformer intensifies exposure bias" is offered as observation without a supporting curve; the 26K / 588K / 0.6M dataset-size numbers appear in adjacent paragraphs without a clear map to which regime each corresponds.

Reproducibility & code

• Fine-tuning subset not released. The 21K PDB fine-tuning subset drives the 161.0 FPSD (against PDB) and 96.6% designability of PARpdb, and FPSD is very sensitive to which structures are in the reference. The paper describes the filtering protocol (re-derived from Geffner et al.) but does not release the PDB-ID list; without it, independent replicas will land on non-identical subsets, and the headline number is not expected to match exactly. The same concern applies to the 26K long-protein PDB subset used in §C.3 / Table 9.
• No random seeds. Neither the training pipeline nor the ablation / evaluation code exposes fixed seeds. Designability and FPSD are computed on 100 samples per length; Monte-Carlo noise for designability at this size is roughly ±3 pp, which is on the same order as several of the reported deltas in Tables 1, 4, and 13.
• Designability pipeline dependencies unpinned. Designability numbers throughout depend on the exact ProteinMPNN and ESMFold checkpoints (only the ProteinMPNN temperature 0.1 is stated). Different public checkpoint versions will shift the numbers.
• NCL noise-weight distribution unspecified. The noise weights $w_{\text{ncl}}^{i}$ are only said to lie in $[0,1]$; the sampling distribution (uniform / beta / etc.), whether they are shared across scales, and the scheduled-sampling probability schedule are not documented.
• Sampling-step configuration under-specified. Table 1 is claimed to use "1k steps" for PAR while Table 8 lists 400/2/2. A reader cannot know which schedule to reproduce.
• Attention-visualization procedure under-specified. The Fig. 6 / Fig. 10 heatmaps require choices — averaging axes over heads/query/key tokens, normalization axis, treatment of the causal / bos structure — that the description does not pin down, and the identity of the 50 test samples is not given.
• Baselines quoted, not recomputed. Baseline numbers other than Proteina are copied from Geffner et al. (2025). FPSD depends on the reference pipeline; unless the same FPSD implementation and reference distribution are used, comparisons in Table 1 are not fully like-for-like.
• Test-set identity for the §C.8 analysis unspecified. "All samples from the testing set" is ambiguous given the multiple test sets used elsewhere; the analysis and the LDDT / RMSE numbers cannot be exactly reproduced without pinning the set.
• Data-split typo. The stated "98:19:1" split sums to 118 and does not normalize; a reader cannot infer whether the intended split was 98:1:1 (standard) or a genuinely unusual 84:16:0.85 partition of 0.6M structures.

Recommended Changes

Essential

• Qualify the 161.0 FPSD claim. In the contribution list, abstract, and Table 1 caption, state that the 161.0 FPSD-vs-PDB is obtained by PARpdb (the 400M model fine-tuned on the re-derived 21K PDB subset), and note that this metric is measured against the same distribution as the fine-tuning data. Report PARpdb's FPSD vs AFDB (228.4) alongside so the trade-off is visible.
• Reconcile the "competitive on FPSD" framing with Table 1. Add a sentence that the like-for-like Table 1 comparison (400M, no PDB fine-tune, default $\gamma$) puts PAR behind Proteina on FPSD, and that the competitive numbers arise at $\gamma=0.45$ or after PDB fine-tuning.
• Reconcile scaling FPSD (187 / 170) with Table 1. Tabulate the raw Fig. 5 values, state the sampling configuration (steps, $\gamma$, SDE vs ODE per scale) and training-step count under which 187 / 170 were obtained, and identify the row of Table 1 they correspond to.
• Fix the Table 3 caption. Either scope "consistently effective" to sc-RMSD / designability or explicitly note that FPSD / fS do not monotonically improve when SS is added. Also fix "effective effective".
• Fix the 98:19:1 typo. State the intended split as e.g. 98:1:1 and update the description.
• Reconcile the sampling-step description in §A.3 with Tables 2 and 8. State the exact per-scale step counts used for every row of Table 1 (and Tables 4, 12, 13).
• Distinguish the 4.7× and 2.5× speed-ups. State each ratio's numerator and denominator inline (which model, which sampling configuration, which hardware).
• Rework §C.8. Either drop the LDDT column (uninformative because saturated) or increase the tolerance sensitivity and report to more decimals; and replace the bicubically-interpolated distance-map reference with the distance matrix of the actual sub-selected $C_\alpha$ points from the coordinate-domain downsampling.
• Release the PDB fine-tuning subset and the long-protein subset (or their PDB-ID lists) plus the filtering scripts, and pin the ProteinMPNN and ESMFold checkpoint versions used for designability. Publish random seeds for training and for the 500-sample designability evaluation, and add per-seed variance (or bootstrap CIs) for the metrics in Tables 1, 4, 5, and 13 where deltas are small.

Suggested

• Add a residue-level continuous-AR baseline of matched decoder capacity to isolate the contribution of the multi-scale factorization, or explain why such a baseline is impractical.
• Add a fine-tuned PAR row in Table 11 so the zero-shot motif scaffolding capability is not conflated with an absolute-performance claim against fine-tuned baselines, and add a sentence stating the trade-off explicitly.
• Address the 4/5-scale non-monotonicity in Table 4. Either soften the "error accumulation" attribution or add a mechanism / analysis (e.g., NCL/SS scaled with $n$) that would predict the observed non-monotone pattern.
• Update the AR-size ablation wording from "does not offer improvements" to "actively worsens", consistent with Table 13.
• Reconcile the 26K / 588K / 0.6M dataset numbers. Add a table (or footnote) listing base-pretrain / PDB-fine-tune / long-protein-fine-tune sizes and clarify which regime the §C.7 exposure-bias argument refers to.
• Add a qualifier to the "reduces to a flow-based model" claim — namely, that at $n{=}1$ the decoder is still conditioned on the learned $\mathbf{z}^{1}$ produced from bos, so PAR reduces to Proteina-with-learned-conditioning rather than to an unconditional flow.
• Specify NCL noise-weight distribution and SS schedule. Add one equation (or one sentence) fixing these choices, so the exposure-bias ablation can be exactly reproduced.
• Specify the attention-visualization reduction and the 50-sample identity so Figs. 6 / 10 can be independently reproduced.
• Align abstract and contribution list on zero-shot generalization. Either soften the abstract to "potential" or strengthen the contribution list to match "strong".
• Soften or substantiate the VAE-latent analogy used to argue that protein modeling is exposure-bias-sensitive.