From Feasible to Practical: Pareto-Optimal Synthesis Planning

Summary

The paper reframes computer-aided synthesis planning from a feasibility problem — "find at least one route to commercial building blocks" — into a multi-objective search problem in which a Pareto front over user-defined objectives is the intended output. Concretely, MORetro* extends the single-objective Retro*/RetroGraph AND-OR search with (i) linear weight scalarization of an $N_D$-dimensional cost vector, (ii) three strategies for sampling the weight simplex (grid, Sobol, and a hypervolume-driven Bayesian-optimization scheme using the GIBBON acquisition function), (iii) a vector-valued lower bound $\mathbf{V}_{\text{bound}}$ that generalizes Retro*'s reaction number to the multi-objective setting and enables safe pruning of frontier molecules, and (iv) three concrete "industrially relevant" objectives — sustainability (temperature + atom economy), toxicity (curated agent tables augmented by GPT-4o and EToxPred), and scale-up potential (ExtractionScore + a MolPrice heuristic). Across USPTO-190, ChEMBL, and Pistachio-reachable with three single-step models (NeuralSym, PDVN, G2E), MORetro* is reported to improve hypervolume by ~5–8%, to dominate 18–36% of baseline solutions while being self-dominated only 2–8%, and to yield small but statistically significant diversity gains, at a success-rate cost of ~1.6–2.1% relative to single-objective Retro*.

Conceptually the move — bringing Pareto reasoning inside the search rather than applying it post-hoc — is timely and industrially motivated, and the $\mathbf{V}_{\text{bound}}$ apparatus is a genuine, cleanly stated contribution. The paper's central conceptual limitation is a tension between what the theory guarantees and what the deployed system does: the completeness argument in Theorem 3.4 assumes admissible heuristics, but the paper's own practical heuristics are inadmissible so the bound collapses to zero in practice, and linear scalarization can in general recover only the convex hull of the front. Combined with a modest empirical margin, a multi-objective baseline that is an untuned self-re-implementation, one of three objectives being ~89% GPT-4o-annotated, and no chemistry-side validation of "practicality," the "true Pareto front" / "from feasible to practical" framing is broader than the evidence strictly supports. The contribution stands, but the headline claims deserve tightening.

Strengths

• Conceptual contribution. Formulating multi-objective synthesis planning as an in-search problem rather than post-hoc filtering, and coupling it with the AND-OR framework, is a genuine and well-motivated advance over the prior best-effort MO-MCTS.
• Optimality apparatus. The vector-valued $\mathbf{V}_{\text{bound}}$ and its role in safe pruning are cleanly derived (Section 3.5, Appendix B); the $\varepsilon$-dominance relaxation recovering an $\varepsilon$-PF is a nice generalization.
• Breadth across single-step models. Reporting on three qualitatively different predictors (template-based NeuralSym, RL-fine-tuned PDVN, template-free G2E) and three datasets usefully demonstrates that the search algorithm is largely independent of the underlying predictor.
• Practical objectives. Sustainability, toxicity, and scale-up potential are more decision-relevant than route length or building-block count, and the QUARC-based condition-prediction integration is a real step toward embedding industrial constraints.
• Ablations reported. Sampling-strategy (BO/Grid/Sobol), pruning (exact vs $\varepsilon$-$\mathbf{V}_{\text{bound}}$), additional fixed-weight configurations, and an MO-MCTS re-implementation are all included, even if imperfect.
• Statistical care. Paired Wilcoxon tests with Bonferroni correction are the right instrument for per-molecule comparisons, and the qualitative Pareto-front structure was confirmed on a single completed target during replication.

Weaknesses

• Scalarization vs. non-convex fronts. The abstract/Section 1 claim that MORetro* "recovers the true Pareto front under admissibility" does not distinguish completeness of the $\mathbf{V}_{\text{bound}}$-driven termination from reachability of the scalarized search. Weighted-sum scalarization cannot reach concave portions of a non-convex Pareto front regardless of sampling density; readers will read a stronger result than the machinery supports.
• Admissibility gap in practice. The Limitations concede that the paper's own heuristics are inadmissible, so the pruning bound falls back to $\mathbf 0$. Correctness is preserved, but the deployed system enjoys no operational benefit from the completeness guarantee — this qualification should sit next to the headline claim, not four pages later.
• No chemistry validation of "practical" routes. HV and R2 measure how well MORetro* optimizes its own proxies; nothing demonstrates that its Pareto-optimal routes are more industrially preferable than Retro*'s. The title/abstract "from feasible to practical" framing rests on this being taken on faith.
• LLM-annotated toxicity objective. ~1,225 of 1,375 agent toxicity scores come from GPT-4o. No agreement statistic against the ~150 curated scores and no ablation removing LLM-scored contributions are reported. As toxicity is one of three Pareto objectives, this noise propagates directly into HV/R2.
• Only one prior MO baseline, and it is a self-re-implementation. The MO-MCTS comparison (Appendix E.3, Table 11) uses the authors' own untuned additive-cost variant. The main-text framing that MORetro* "outperforms multi-objective baselines" is largely underwritten by this untuned comparison.
• Scalability to $N_D > 3$ untested. Every experiment uses three Pareto objectives, yet simplex sampling, HV computation, and GP-based BO all scale poorly with $N_D$. The "any number of user-defined objectives" claim is unsubstantiated beyond the tested regime.
• Normalization/reference point underspecified. HV/R2 use 5th/95th-percentile normalization with reference point $[1.1, 1.1, 1.1]$, but the paper does not state whether percentiles are pooled across methods, per-method, or per-molecule, nor how out-of-range costs are clipped. Given ~0.05 headline HV gaps, these choices are of the same order as the effects.
• No per-seed variance. All error bars are over molecules, not repeated runs; the BO/Sobol sampler is stochastic, so it is unclear whether the 5–8% HV gain exceeds run-to-run noise.
• Text/table inconsistencies. "2–3 additional molecules" vs. Table 2 (~3–4); "up to 25% fewer" vs. Table 2's -29.1%; Figure 4 "visually confirming the significance" conflates illustration with the Wilcoxon test; the $\varepsilon$-$\mathbf{V}_{\text{bound}}$ "without deteriorating" framing hides a small R2 regression vs. exact $\mathbf{V}_{\text{bound}}$.
• Objective definitional gaps. The scale-up split between running separation cost and the MolPrice future heuristic is not made explicit; MolPrice normalization units ("0–15") are unstated; the separation penalty $C(P_{\text{diff}})$ shape is not given; the guidance-heuristic rationale for RL-trained models conflates "policy is good" with "zero heuristic is safe."

Reproducibility & code

The code was executed. The core search engine (moretro/moretro_star.py, moretro/search/mo_search.py, mo_graph.py, the BO/grid/Sobol samplers, and the pruning dominance check) ran essentially as shipped, with only minor behavior-preserving edits, and a single completed G2E USPTO-190 target produced a well-formed 3-objective Pareto front (31 non-dominated routes separated from 50 dominated), matching the structure of Figure 1. That is the extent of what reproduced.

• Headline aggregates did not reproduce. 18 of 19 tracked claims are "not attempted." On a single pinned CPU core one target took ~30 minutes and the wall-clock elapsed before the metric-aggregation stage ran, so no cell of Tables 1, 2, 3, or 5 (HV/R2/dominance/success/diversity) was produced. This is largely environmental (the paper documents RTX6000 / EPYC 7742 hardware), but it is compounded by the code gaps below, and the ~5–8% HV improvement therefore remains unconfirmed by execution.
• Evaluation harness absent from the repository. There is no HV/R2 aggregation script applying the 5th/95th-percentile normalization and $[1.1,1.1,1.1]$ reference point, and no R2 (Brockhoff et al., 2012) implementation at all; the reproducer had to author these from scratch, and — because aggregation never ran on real output — the reimplementation is itself unvalidated.
• Diversity metric missing. The Genheden & Shields (2025) route-similarity metric behind Tables 3/7 is not implemented, though the input route JSONs are saved.
• Baseline runners missing. The Fixed baseline is reachable only via a constant-weight hook hard-coded to $[0.2,0.2,0.2,0.4]$; there is no named Retro* single-objective runner. Reproducing Table 1's Retro* columns and Table 8's fixed-weight sweep required source patches and hand-written configs.
• Budget mismatch. The shipped config sets the single-step budget to 350, whereas the paper states $N_B = 300$; Table 4's model-call statistics (289 / 265) sit near that boundary.
• Statistical, pruning-analysis, and MO-MCTS code missing. No Wilcoxon/Bonferroni script, no harness producing Table 4's cut counts and pruned-vs-unpruned deltas, and no additive-cost MO-MCTS variant (Table 11) are present in the repository.
• Missing inputs and underspecified setup. None of the USPTO-190, ChEMBL-150, or Pistachio-reachable target lists is shipped (only a 2-molecule example), so all three had to be reconstructed; the BO warm-up grid resolution is not enumerated. Measured per-target runtime (~30.5 min) is ~2.5–3x the stated 6–12 min, consistent with the single-core constraint on non-comparable hardware rather than a substantive contradiction.

The algorithmic core is present and runnable, but the entire evaluation layer that turns raw routes into the paper's headline tables is user-supplied, and none of it was confirmed against the paper's numbers in this run.

Recommended Changes

Essential

• Tighten the "recovers the true Pareto front" claim. In the abstract, introduction, and Section 3.5, separate (a) the completeness argument of Theorem 3.4 under admissibility from (b) the convex-hull limitation of linear scalarization; state plainly that concave portions of the front are not guaranteed reachable by weight sweeping.
• Move the admissibility caveat to the headline claim. State next to the guarantee that the practical heuristics are inadmissible, so $\mathbf{V}_{\text{bound}}$ collapses to $\mathbf 0$ and Theorem 3.4's pruning benefit is not operational in the deployed system; point to Table 4 as the realized measure.
• Release and validate the evaluation harness. Ship (a) the USPTO-190 / ChEMBL-150 / Pistachio-reachable target SMILES lists, (b) the HV/R2 aggregation script with the paper's normalization and reference point plus an R2 implementation, (c) the Genheden & Shields (2025) diversity metric, (d) explicit, configurable Retro* and Fixed baseline runners, (e) the Wilcoxon/Bonferroni script, (f) the pruning-analysis harness behind Table 4, and (g) the additive-cost MO-MCTS variant. Align the shipped budget to $N_B=300$ or document the 350 setting, and add a lightweight/multi-core preset so the headline tables can be reproduced within a normal wall-clock.
• Validate the LLM-annotated toxicity objective. Add agreement statistics on the curated/GPT-4o overlap, an ablation zeroing the LLM-scored contribution, and at minimum a chemist spot-check on a random sample of predicted scores.
• Add a chemistry-side evaluation of "practicality." Provide at least a small expert-ranking case study of MORetro* vs Retro* Pareto routes to support the "from feasible to practical" framing beyond proxy optimization.
• Reframe the MO-baseline claim. State explicitly in Section 4 that the only MO comparison is an untuned self-re-implementation with a modified reward, and soften "outperforms multi-objective baselines" accordingly; if feasible, add the original terminal-objective MO-MCTS on a common objective subset at matched budget.
• Pin down HV/R2 normalization. Specify whether percentiles are pooled, per-method, or per-molecule; state the clipping convention for out-of-range costs; add a robustness check with normalization fitted on Retro* alone.

Suggested

• Report per-seed variance. Run MORetro* with ~3 seeds on a USPTO-190 subset and report seed-vs-molecule variance so readers can judge whether the 5–8% HV gain exceeds sampler noise.
• Test $N_D > 3$. Add an ablation at four or five objectives, or narrow the "any number of objectives" claim to the tested regime.
• Reconcile text with Table 2. Update "2–3 additional molecules" to "3–4" and "up to 25% fewer" to "up to ~29% fewer."
• Soften Figure 4 language. Replace "visually confirming the significance" with "illustrating the improvement," and state the baseline HV so the "~10%" follows from $\Delta \mathrm{HV}=0.07$.
• Clarify the $\varepsilon$-$\mathbf{V}_{\text{bound}}$ trade-off. State explicitly that it trades a small R2 regression vs. exact $\mathbf{V}_{\text{bound}}$ for extra runtime savings.
• Fill in objective definitions. Give the closed form of $C(P_{\text{diff}})$; specify the units of the "15" MolPrice bound; make explicit that MolPrice is the $\mathbf{V}_m$ heuristic (not a running cost) for scale-up; and separate the "policy is good" from the "zero heuristic is safe" argument for the guidance heuristic.
• Specify the BO warm-up grid. State the coarse-grid resolution and enumerate the 10 warm-up points satisfying $w_{N_D} \ge 0.5$.