OMAC: A Holistic Optimization Framework for LLM-Based Multi-Agent Collaboration

Summary

The paper introduces OMAC, a framework for optimizing LLM-based multi-agent systems (MAS) across five explicitly enumerated dimensions — two functional (existing-agent prompts, new-agent construction) and three structural (candidate selection, dynamic per-step participation, and communication routing). The conceptual move is to view a MAS as an information-flow graph and argue that optimizing the graph's nodes and edges reduces to optimizing prompts for either agents (nodes) or LLM-powered controllers (edges/selection). Two LLM-powered actors — a Semantic Initializer that seeds a diverse collection of prompts and a Contrastive Comparator that reasons over sampled positive/negative pairs — power a per-dimension loop, and a fixed-one-at-a-time iterative schedule scales that loop to joint optimization. Experiments on HumanEval, MMLU, and MATH (plus MBPP and GAIA in the appendix) use DyLAN's MAS as the starting configuration and report OMAC beating single-agent, fixed multi-agent, and MAS-optimization baselines (DyLAN, ADAS, AFlow) across three headline tasks and four base LLMs. Ablations indicate the Comparator adds over the initializer alone, and sensitivity studies suggest the method is not knife-edge in its hyperparameters. This is a useful synthesis paper: the dimension taxonomy is a genuinely helpful teaching device, and using LLM contrastive reasoning as the mutation operator inside an evolutionary-style loop is a clean idea. The main conceptual limitation is that "holistic"/"comprehensive" coverage is asserted via an informal graph analogy rather than proven, and step count, edge weights, memory/retrieval, and stopping rules sit outside the taxonomy. Empirically, gains are often small relative to the reported variance on small test sets, and the headline figures are stated inconsistently and without a clearly specified reference baseline. Executing the released code strengthens confidence in the pipeline and the directional trend (one HumanEval cell matched within 1σ; MATH/most-MMLU dimensions exceed the quoted baselines), but no baseline was independently re-measured, the shipped splits are smaller than the paper's, runs were single-seed on a different model snapshot, and the largest claimed gains — multi-dimension joint optimization — were not reproducible.

Strengths

• Clear taxonomy. The five-dimension decomposition — two functional (Fun-1 existing agent, Fun-2 new agent) and three structural (Str-1 candidate selection, Str-2 dynamic participation, Str-3 communication routing) — is a usefully organized lens, and Sections 3.1 and B make the graph analogy explicit enough for other work to build on.
• Unified algorithm across dimensions. The same Semantic Initializer / Contrastive Comparator pair applies to all five dimensions with only contextual changes, which is more elegant than the current landscape of one-off, per-dimension methods.
• Ablation that isolates the Comparator. OMAC-C (initializer only) is a well-designed ablation and consistently underperforms full OMAC across every dimension in Tables 4/8/9, giving concrete evidence that contrastive refinement — not just diverse seeding — drives the gains.
• Breadth of empirical coverage. Three main benchmarks plus MBPP and GAIA, four base LLMs, and hyperparameter sweeps over collection size, iteration count, and sampling thresholds make the empirical footprint substantial for a MAS optimization paper.
• Inference-cost discussion. Reporting API-call counts and dollar cost per problem (Tables 10/11/13) is welcome and unusually concrete; the structural variants do appear to reduce cost relative to DyLAN and AFlow.
• Pipeline validated by execution. Re-running the released HumanEval loop end-to-end reproduced the Programmer (Fun-1.4) cell at 88.75% versus the paper's 89.25 ± 1.30 — a match within one standard deviation that confirms the full Initializer→collaboration→Comparator→unit-test path works as described.

Weaknesses

• "Holistic"/"comprehensive" overclaimed. The node/edge analogy is intuitive, not exhaustive: step count and ordering, edge weights beyond a binary route/no-route decision, memory/retrieval, and stopping rules are all information-flow aspects the five dimensions do not cover. The abstract's "holistic" inherits the same overreach.
• Statistical support is thin for the central claim. Many gains are within the reported standard deviations (e.g., HumanEval Str-1 86.76 ± 1.22 vs DyLAN 85.74 ± 2.83), test sets are small, only three repeats are run, and no significance test is reported, so "significantly/consistently outperforms" outruns the evidence.
• Validation split defined but unused. Data are split 3:1:1, yet best-configuration selection is done on the training set while the validation split is apparently never used — risking optimistic selection given the high variance.
• Headline gain figures are inconsistent and under-specified. Section 4.1 reports per-dimension averages of 3.6%/2.8%/4.9%; Appendix D.2 reports 4.6%/2.8%/5.3%. Neither states the reference baseline, and the "2.9% to 9.6%" joint figure is ambiguous between absolute and relative and is read off a figure with no numeric table.
• Truncated Functional-dimension columns. Tables 1 and 2 (and 14/15) as rendered show only Structural columns, yet the headline averages are computed over Fun-1.x and Fun-2 — the reader cannot verify them.
• Cross-model transferability overclaim. Section C.2.5 re-optimizes per model; it does not show optimized prompts transfer across models, which is the stronger claim the wording implies.
• Metric and scope slips. GAIA is labeled "Pass@1" though the paper defines Pass@1 via unit tests; HumanEval is framed as tool-use evidence though it is code completion; the inference-cost claim is generalized past the structural dimensions that actually prune agents/edges.
• "Distinct paradigm avoiding search" vs. an evolutionary loop. OMAC maintains a population, scores by fitness, selects by rank thresholds, and re-injects a mutated candidate — structurally a guided evolutionary loop; the real novelty is the LLM contrastive-reasoning mutation operator.
• Unsupported specific numbers. The three-dimension "5%"/"1.8×" figures and the dimension-order convergence claim in D.2 have no supporting table or figure.
• Whole-set vs. 50%-subset framing conflict. D.1/D.5 assert whole-training-set validation while C.2.4 promotes a 50% subset as the default; Tables 12/13 appear to be from the subset regime without labeling.
• Notation and minor errors. Definition 2 conflates a step $s_t \in S$ with a set of agents $\{a_{t,\cdot}\}\subseteq s_t$; "exponential" mislabels a multiplicative cost; the Table 6→7 cross-reference is wrong; Table 10 (0.1958) contradicts Table 13 (0.1928) for AgentVerse; the Str-3 "another Tester agent" example does not exist in the described MAS; "well-optimized and near-optimal" mischaracterizes AgentCoder/ReAct; and there is an "on the on the" typo.

Reproducibility & code

Veritas executed the released repository (HumanEval, MMLU, MATH) rather than only inspecting it; the overall reproduction score is 0.5, with the three headline table claims graded partial and the multi-dimension and supporting claims not reproduced.

• Pipeline runs but is fragile. The HumanEval loop ran end-to-end and reproduced the Fun-1.4 cell (88.75% vs 89.25 ± 1.30, within 1σ), but only after fixing a scorer/subset mismatch and a multiprocessing-sandbox bug in the shipped code. The artifact is runnable-but-fragile, not turnkey.
• Directional trend holds; strict ordering does not. On MATH all six reproduced OMAC dimensions (38.1-41.9%) clearly exceed the quoted AFlow/DyLAN baselines; on MMLU most dimensions do too. But two MMLU dimensions (Fun-1.3, Fun-1.6 = 67.74%) fall below AFlow's 70.06%, and the exact-model MATH Str-1 (30.95%) dips below both baselines, so the categorical "all dimensions beat the strongest baseline" is not reproduced.
• No baseline re-measured. The repo ships no baseline implementations or pinned configs, so every baseline number in Tables 1-3 is paper-quoted; the OMAC-vs-baseline gaps could not be independently verified against stochastic small-set runs.
• Shrunken splits and single seed. The bundled MMLU test split is 31 questions (coarse 1/31 = 3.23pp grid), not the paper's 84, and the MMLU arm ran at a single seed despite the stated three-repeat protocol — so the reported means/σ cannot be reproduced at the paper's resolution.
• Model-snapshot sensitivity. The primary arm used gpt-3.5-turbo-0125, which inflated MATH accuracies 2-8pp above the paper; re-running on the paper's gpt-3.5-turbo-1106 agreed much more closely (mean 35.16 vs 34.08). Per-cell magnitudes are snapshot-sensitive.
• Biggest claim not reproducible. The multi-dimension joint optimization behind the "2.9% to 9.6%" headline could not be reproduced within budget, and Figures 4-6 have no underlying numeric table to check.
• Whole appendix arms absent. No MBPP loader, no GAIA tool-use stack, and no multi-LLM adapters (the model layer raises NotImplementedError beyond gpt-3.5), so Tables 14-18 could not be attempted. The OMAC-C ablation is not exposed as a switch, so Tables 4/8/9 require a manual code edit.

Recommended Changes

Essential

• Report significance and use the validation split. Add a paired significance test (or bootstrap CIs) across seeds for the headline OMAC-vs-baseline comparisons, and perform best-configuration selection on the held-out validation split rather than the training set (see Statistical support and Validation split in Weaknesses).
• Reconcile and ground the headline gains. Fix the 3.6%/2.8%/4.9% vs 4.6%/2.8%/5.3% mismatch, state the baseline entering the "relative gain" denominator, and convert "2.9% to 9.6%" into absolute pre/post accuracies with variance; add a numeric table behind Figures 4-6 (see Headline gain figures and the multi-dimension repro point).
• Ship a verifiable artifact. Fix the scorer/subset and multiprocessing-sandbox bugs, pin the exact model snapshot (gpt-3.5-turbo-1106) and data splits, expose seed/repeat settings, and add a smoke test, so the pipeline and reported means/σ are reproducible (see Pipeline runs but is fragile, Shrunken splits, Model-snapshot sensitivity).
• Pin and ship baselines, or scope the claims. Provide the baseline implementations/commit pins used for Tables 1-3, or explicitly mark those numbers as taken from prior work; qualify the ordering claim to match the reproduced "in most cases" behavior (see No baseline re-measured and Directional trend holds).
• Restore the Functional-dimension columns in Tables 1, 2, 14, 15 so the reported averages can be verified (see Truncated Functional-dimension columns).
• Add the missing appendix harnesses and an ablation switch. Ship MBPP/GAIA harnesses and multi-LLM adapters (or document the edits), and add a "--no-comparator" mode so Tables 4/8/9 and 14-18 are reproducible (see Whole appendix arms absent).

Suggested

• Weaken "holistic"/"comprehensive" to node-and-edge coverage of the collaboration graph, and list what is out of scope (step count/ordering, edge weights, memory, stopping) — see "Holistic"/"comprehensive" overclaimed.
• Relabel the GAIA metric as accuracy/exact-match and restrict tool-use evidence to GAIA, describing HumanEval as code generation (see Metric and scope slips).
• Scope the inference-cost claim to structural-dimension configurations (see Metric and scope slips).
• Downgrade "cross-model transferability" to "generalizes across base LLMs when applied per model," unless a true transfer experiment is added (see Cross-model transferability overclaim).
• Soften "distinct paradigm avoiding search" to name the LLM contrastive-reasoning mutation operator as the contribution (see the evolutionary-loop weakness).
• Back or delete the specific numbers for the three-dimension "5%"/"1.8×" and dimension-order claims; reconcile the whole-set vs. 50%-subset framing and label which tables use which (see Unsupported specific numbers and Whole-set vs. 50%-subset).
• Fix notation and nits. Introduce $A_t \subseteq A$ for the step-$t$ agent set; replace "exponential" with "multiplicative"; correct the Table 6→7 reference; reconcile the Table 10 vs 13 AgentVerse cost and the OMAC (avg.) 0.0974 basis; fix the Str-3 "Tester" example; retone "well-optimized and near-optimal"; and remove the "on the on the" typo (see Notation and minor errors).