FLIP2: Expanding Protein Fitness Landscape Benchmarks for Real-World Machine Learning Applications

Summary

FLIP2 is a substantial expansion of the FLIP protein-fitness benchmark. The paper adds seven new sequence-fitness landscapes (alpha amylase, imine reductase, nuclease B, TrpB, a three-scaffold hydrophobic-core dataset, microbial rhodopsins, and a PDZ3/CRIPT interaction assay) and 16 splits grouped into five extrapolation regimes — number-of-mutations, position, mutation identity, fitness ranking, and wild-type / scaffold shift. The conceptual move is to shift emphasis from "how well can a model interpolate on a random split?" to "does a model generalize across the specific distribution shifts that arise in a real engineering campaign?", and to widen functional coverage beyond thermostability, binding, and viral viability. The evaluation compares three families of baselines that bracket what the field typically deploys: zero-shot pLM likelihoods (Dayhoff, CARP-640M, ESM2-650M, SaProt-650M), ridge regression on one-hot sequences with and without pLM-likelihood augmentation, and fine-tuned pLMs (CARP-640M, ESMC-300M, SaProt-650M, both full and LoRA). The headline finding — that simple models remain competitive with fine-tuned pLMs, and that transfer to new wild types and new positions is where all methods break down — is consistent with concurrent literature and is the kind of negative result the community needs.

An independent re-execution of the pipeline supports the paper's central story: the "simple models match or beat fine-tuned pLMs" result reproduces with large margins on most covered splits, and "FLIP2 splits are harder than random splits, hardest on wild-type/position/fitness" reproduces cleanly across all 16 splits. The main conceptual limitation is that "simple beats fine-tuned pLM" is derived from an argmax over per-split point estimates with only five fine-tuning seeds, no significance tests, and a categorical partition whose boundaries are load-bearing but loosely defined. Several claims also disagree across sections (wild-type splits in particular), the zero-shot taxonomy mis-buckets PDZ3, and some Table A1–A16 rows are numerically impossible (NDCG > 1) or byte-for-byte identical between structure and sequence variants. Reproducibility is partial and fixable: the released code did not run as shipped.

Strengths

• Conceptual contribution. Framing benchmark splits around real engineering distribution shifts (number, position, mutation, fitness, wild-type / scaffold) rather than i.i.d. splits is the right move for the field, and the five-way taxonomy is clean and reusable.
• Functional diversity of datasets. The seven new landscapes cover under-benchmarked regimes: enzymatic activity on small molecules (Amylase, IRED, TrpB), nucleic-acid substrates (NucB), hydrophobic-core stability across three scaffolds, IDR-mediated PPI (PDZ3/CRIPT), and light-sensitive proteins (Rhomax). This meaningfully broadens FLIP's original coverage.
• Baseline design. Explicitly contrasting zero-shot likelihoods, one-hot ridge, ridge augmented with zero-shot scores, full fine-tuning, LoRA, and a naive-supervised (random-init) control is exactly the set of comparisons needed to isolate the value of pretraining versus supervision.
• Headline findings are reproducible. Independent re-execution confirms that ridge with one-hot + pLM likelihoods matches or beats the best fine-tuned pLM on ~8/10 covered splits (large margins on amylase and rhomax), and that FLIP2 splits are uniformly harder than matched random splits, with wild-type/position/fitness the hardest. The qualitative claims hold up beyond the authors' own reporting.
• Data provenance and licensing. Redistributing all data under CC-BY 4.0 with recorded provenance removes a real friction for follow-up work.

Weaknesses

• Statistical robustness of the headline count. The "7/4/4/1 winners across 16 splits" tally is an argmax over point-estimate Spearmans with only 5 seeds and no significance test. Per-table stds are often as large as the winner-runner-up gap (NucB ESMC-300M LoRA 0.730±0.008 vs supervised 0.723±0.002; Hydro to-P0A9X9 winner 0.171; Amylase one-to-many CARP supervised 0.242±0.153). The categorical partition is also underspecified: "naive supervised" is counted apart from "fine-tuned," though both are supervised pLM training.
• Overclaim in the abstract. "Challenging the utility of existing transfer learning techniques" is broader than Table A18 supports (a fine-tuned pLM still wins 7/16 splits, including IRED, NucB, PDZ3). The reproduced result is best described as "competitive, especially on wild-type and position extrapolation."
• Contradiction between Results and Discussion on wild-type splits. Section 4.3 says supervised pLMs "usually outperformed ridge regression with both sequence and zero-shot likelihoods for wild type and number splits," while the Discussion says fine-tuned pLMs "do not consistently improve over simpler methods on wild-type and position splits." Table A18 supports the latter for three of four Hydro wild-type splits.
• Muddled scoring taxonomy. PDZ3 is listed among "variants of a single wild type" for the mask-and-sum log-likelihood-ratio recipe, yet is explicitly a two-protein interaction with a special two-sequence rule one sentence later; and Section 4.1's "variants derived from in multiple (Hydro, Rhomax)" is grammatically broken.
• Ambiguous pseudolikelihood definition. "Pseudolikelihood of each entire sequence with no positions masked" is not the standard mask-in-turn convention for MLMs and is not the right term for autoregressive Dayhoff; reproducers cannot tell which score was computed for Hydro and Rhomax.
• SaProt announced but partially excluded. SaProt is one of "four representative pLMs," but Figure 2, the ridge-with-likelihoods baseline, and the "no single pLM is optimal" summary use only the other three, and the pretraining-improvement statistic is not reported for SaProt despite its being chosen to test structure tokens.
• Suspect table entries. NDCG > 1 for SaProt likelihoods on IRED (1.451/1.436), PDZ3 (1.153/1.159), and several Hydro rows contradicts the definition of NDCG; multiple SaProt structure/sequence rows are byte-for-byte identical across five seeds (NucB A6, Amylase A1/A2/A4, Hydro A10/A11); Table A9 reports a ± nan std with no explanation.
• No noise ceiling. Without replicate correlation or an equivalent per-dataset ceiling, a Spearman of 0.6 (Amylase) or 0.5 (PDZ3) cannot be judged near-saturated or far from ceiling.
• No pretraining-overlap control. All FLIP2 wild types are natural proteins almost certainly in the pLM pretraining corpora; the pretraining-vs-supervision interpretation does not acknowledge or quantify this.
• Uneven split coverage. Only Amylase covers four split types; IRED, NucB, and PDZ3 each contribute one number-split, so aggregate claims about position/backbone generalization rest on very few splits.
• Fixed ridge regularization. Ridge is the counter-example that carries the headline; a validation-tuned penalty or a sensitivity sweep would remove the concern that a single fixed value distorts the data-size comparison.
• Missing bridge to prior benchmarks. No baseline is run on the original FLIP splits or ProteinGym supervised subsets, so "FLIP2 is harder" is not shown apples-to-apples.
• Minor issues. Boilerplate dual-use paragraph; Hydro framed without noting it samples ~10% of the combinatorial space; NucB bins not exhaustive at exact wild-type/top-mutant activity; naive-supervised CARP winning Hydro three-to-many is left undiscussed.

Reproducibility & code

The code was re-executed for this review; where the release could not run, the missing pieces were reconstructed from the manuscript. Overall the paper earns a strong replication score: the headline conclusions hold, but the release is not runnable as shipped.

• Headline results reproduce. Reconstructing Figure 4C, ridge (one-hot + pLM likelihoods) matches or beats the best fine-tuned pLM on ~8/10 splits that had a completed fine-tuning run, with large margins on amylase (e.g. 0.735 vs 0.431) and rhomax. Reconstructing Figure 3, the FLIP2-split ridge Spearman is below the matched random-split value for all 16 splits, with the largest drops in wild-type/position/fitness. Zero-shot likelihoods are clearly more predictive on single-wild-type datasets (group mean ≈0.43) than on multi-wild-type (≈0.09) or PDZ3 (≈0.13). These match the paper.
• Pretraining-benefit counts only partially reproduce. The exact 14/16 (CARP) and 9/16 (ESMC) counts were unreachable because full-parameter fine-tuning across all 16 splits x 2 inits was too expensive on the available GPUs; only 7/16 (CARP) and 9/16 (ESMC) pairs completed. The direction and fractions align (CARP 86% vs 88%; ESMC 67% vs 56%), and pretraining hurts the rhomax wild-type split for both models, but the headline counts themselves were not confirmable at full coverage.
• Release did not run as shipped. Executing the pipeline required nine fixes, six major, none critical. Only the CARP/ESMC fine-tuning drivers ran as shipped; the masked zero-shot scorer crashed (expects a variant_info column absent from the released CSVs); the ridge driver had hard-coded personal paths (/home/kevyan/...) and expected pre-merged inputs; the dependency's load_flip_data mishandles FLIP2's boolean validation column and would silently corrupt splits; and no FLIP2 aggregation/figure code ships at all. The two fair-esm / ESM-C environments cannot coexist (shared esm namespace) — undocumented. The printed URLs contain stray whitespace.
• IRED total does not match the release. Table 1 lists IRED n_total = 17,143, but the released split CSV exposes only 8,586 variants; the other six datasets match Table 1 exactly or within rounding. The IRED split is a subset of the full library, undocumented in text or artifacts.
• Table 1 internal inconsistency for NucB. The NucB row states 55,760 but its train+val+test sum to 57,313 (and a raw re-count gave 55,759).
• SaProt/Dayhoff zero-shot and all LoRA variants unverified. No runnable code ships for SaProt structure-token derivation from the Table A17 PDB codes or for the LoRA loop, so the Table A18 winners attributed to SaProt likelihood (Hydro to-P06241), SaProt supervised (Hydro to-P0A9X9), and ESMC-300M LoRA (IRED, NucB, Hydro to-P01053) cannot currently be reproduced.

Recommended Changes

Essential

• Add a paired significance test for the winner tally. Replace the argmax-over-mean count with a paired bootstrap over per-split test predictions (or a Wilcoxon signed-rank across the 16 splits) comparing ridge+likelihoods vs the best fine-tuned pLM, and define which model families are grouped into "fine-tuned." This addresses the statistical-robustness and category-boundary weaknesses.
• Reconcile the wild-type-split contradiction. State clearly whether supervised pLMs beat ridge+likelihoods on wild-type splits (Section 4.3) or do not (Discussion), and align Table A18 and Figure 4C with the correct direction.
• Scope the abstract's second clause. Change "challenging the utility of existing transfer learning techniques" to reflect that simpler models are competitive especially on wild-type and position extrapolation, matching Table A18 (fine-tuned pLMs win 7/16).
• Make the release runnable and document coverage. Parameterize the hard-coded /home/kevyan/... paths; ship the SaProt zero-shot scorer (with structure-token derivation), the SaProt fine-tuning driver, the LoRA loop, and the FLIP2 aggregation/figure code; document the two-environment requirement; and report the fine-tuning seed count and compute budget behind the 14/16 and 9/16 statistics.
• Fix the numerical artefacts in Tables A1–A16. Diagnose and correct or drop the NDCG > 1 entries; identify why SaProt structure/sequence rows are identical and re-run whichever variant was not executed; explain the ± nan std in Table A9.
• Reconcile Table 1 with the released data. Document how the full IRED library (17,143) reduces to the released split (8,586), or release the full table; correct the NucB total so its split sizes sum consistently.
• State the exact scoring convention for Hydro and Rhomax, and move PDZ3 out of the "single wild type" list, describing its two-sequence rule (and the aggregation) in one place.

Suggested

• Add a per-dataset noise-ceiling estimate (replicate correlation, or best-case ranking of the NucB bins) so reported Spearmans can be read against a ceiling.
• Report per-family training-corpus overlap (e.g. minimum identity to any UniRef50 cluster) so the pretraining-benefit interpretation is qualified by exposure.
• Add a ridge regularization sensitivity sweep (or a validation-selection description) to show the "ridge is competitive" result does not depend on one fixed penalty.
• Bridge to prior benchmarks by running ridge (one-hot + likelihoods) and CARP/ESMC supervised on FLIP AAV/GB1/thermo and a few ProteinGym DMS assays.
• Soften the position-generalization causal claim ("fine-tuning hurts generalization to different positions") to an observation over three splits, and add per-split-type winner counts alongside the aggregate tally.
• Add per-model standard deviations to Table A18 and a confidence interval on the Rhomax winner.
• Add one sentence discussing where pretraining hurts CARP (e.g. Hydro three-to-many, where naive-supervised CARP wins), note that SaProt-structure uses a single wild-type PDB even for far-from-wild-type variants, make the NucB bin edges exhaustive, state that Hydro subsamples the combinatorial space, and replace the generic dual-use paragraph with a concrete statement of included/excluded protein classes.
• Fix the two URLs in Software and Data (flip.protein. properties, zenodo. 18433203).