Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning

Summary

Agent0-VL proposes a self-evolving vision–language agent that unifies a Solver (multi-turn tool-integrated reasoning) and a Verifier (tool-grounded generative critique with step-level score/confidence/critique tuples) inside a single LVLM sharing parameters $\theta$. A confidence-gated Self-Repair module rewrites low-quality segments, and the two roles are jointly optimized by GRPO under a composite reward that combines an outcome term, a Verifier-derived process term, tool and cross-role regularizers, and a repair cost — the Self-Evolving Reasoning Cycle (SERC). The conceptual move is real and worth publishing: extending tool integration from reasoning to self-evaluation and self-repair is a natural next step for self-rewarding VLMs, and the argument that text-only self-critique is unreliable on visually grounded math/geometry problems is well-motivated. Reported gains are large across seven benchmarks (Table 1), the ablation ordering is clean (SERC > Tool Use > Self Repair contribute in that order), and the PRM transfer result (Table 4) suggests the Verifier learns something usable outside the coupled loop. Two structural concerns pull against these strengths. First, the "zero-external-reward" framing is materially inaccurate: an explicit external-reward RL warm-up (Appendix B) and GPT-5 / Qwen2.5-VL-72B teacher distillation for the 200k SFT corpus both inject supervision that the abstract disavows; the paper should describe the contribution as tool-grounded self-improvement on top of an externally-shaped policy. Second, the self-referential optimization (policy scores itself, policy is trained on those scores) admits a straightforward reward-hacking failure mode that the paper acknowledges only through a $\beta_{\text{div}}$ coefficient with no diagnostic of Verifier calibration or drift across iterations. On top of these, the manuscript has extensive presentation issues — missing display equations for the central reward machinery, notation collisions ($g_t$ vs $g(\tau)$; RA/EE vs S/V; $\eta$ vs $C_{\text{repair}}$; $N$ vs $G$), an undefined benchmark (MME-Real), a mislabeled Table 4 row, and case-study figures whose tool outputs actively contradict the tool-grounding thesis. The idea is strong; the empirical and technical presentation still needs consolidation.

Strengths

• Conceptual contribution. Elevating tool use from a reasoning aid to a verification and repair primitive is a coherent extension of the self-rewarding VLM line; the argument that tool-grounded critique is more reliable than text-only reflection on visually grounded problems is well-motivated in Section 1 and pays off in the ablation where removing tool use drops Math-avg by ~6.5%.
• Cleanly ordered ablation. Table 3 shows the expected ordering full > w/o Self-Repair > w/o Tool Use > w/o SERC > Base on Math Avg, and the biggest drop is where the paper argues it should be (removing the outer SERC loop). This is stronger evidence for the framework than the headline number alone.
• PRM transfer result. Deploying the Verifier as a PRM for other LVLMs (Table 4, Figure 3) is a genuinely informative decoupled test — it shows the Verifier learns a reusable scoring signal, not just an internal reward that overfits its own Solver.
• Two base-model families. The paper reports Agent0-VL on both Qwen2.5-VL-7B and Qwen3-VL-8B (Table 1), which mitigates the concern that the method is a Qwen2.5-specific trick.
• Detailed appendix. Appendix B (hyperparameters, hardware), Appendix D (data construction with quality-control passes), and Appendix E (system prompts, worked case studies) give reviewers more to work with than most papers of this scope.

Weaknesses

• Zero-external-reward claim contradicts Appendix B. The abstract says the model self-evolves "without any human annotation or external reward models"; Appendix B describes a two-epoch external-reward RL warm-up "before self-evolution begins" that the authors credit with preventing "early-stage collapse or entropy degeneration". The self-evolving phase therefore sits on a policy already shaped by external correctness signals — this needs to be reflected in the abstract/contribution and in the Related Work framing against methods that also warm-start with external rewards.
• SFT teacher distillation is external supervision. The 200k SFT corpus is generated by GPT-5 and Qwen2.5-VL-72B (Section 5.1). Because those teachers are themselves aligned with human feedback, the reasoning/critique/repair formats — and their preferences — enter Agent0-VL indirectly. "No human annotation" is a strong claim; please rephrase to describe what supervision is absent at the self-evolving stage rather than what the whole pipeline foregoes.
• Reward-hacking risk is unaddressed. The Verifier is a mode of the policy being trained on rewards produced by that Verifier. The paper mentions a "cross-role regularization" coefficient $\beta_{\text{div}}$ but never shows the divergence term, its target distribution, or any diagnostic (e.g., Verifier accuracy against an external oracle, ECE of $\text{conf}_t$, or agreement rate between Verifier and $r_{\text{out}}$) across iterations. Without such diagnostics, monotone gains in Table 2 are compatible with the Verifier learning to reward itself.
• Missing display equations for the central reward machinery. Four key definitions — the Verifier tuple $V_t$, the process reward $r_{\text{proc}}^{(t)}$ (Eq. 2), the repair gate $g_t$ (Eq. 3), and the effective step reward (Eq. 4) — are introduced by "where …" clauses without the preceding equation. The reader can see the coefficients ($\lambda_{\text{tool}}$, $\beta_{\text{div}}$, $\kappa$, $C_{\text{repair}}$) but not how they combine, so the reward function is not specified as written.
• Confidence-gating direction is inconsistent. Algorithm 1 fires repair when $\text{conf}_t < \tau_c$; Figure 8 fires it on "$\text{conf}_t \ge \tau_c$ and $\text{score}_t < 0$"; Section 5.5 describes "high-confidence feedback" triggering repair. Two different meanings of $\text{conf}_t$ (Solver certainty vs. Verifier certainty in a negative critique) are collapsed into the same symbol and threshold.
• Sigmoid gate vs hard threshold. Section 4.2 defines the gate through $\sigma(\cdot)$ and a temperature $\kappa$; Algorithm 1 uses a hard comparison. The operational form used in experiments is not stated.
• Notation collisions. $g_t$ (per-step gate) collides with $g(\tau)$ (trajectory return); Algorithm 1 uses role tokens "RA/EE" that are never mapped to the defined $m \in \{S, V\}$; Appendix B introduces a repair penalty $\eta = 0.05$ that never appears in any equation, while the main text uses $C_{\text{repair}}$; GRPO group size $N=8$ coexists with "4 rollouts per task" without either being reconciled with the formal group size $G$ in Eq. 6.
• ChartQA improvement figure does not reconcile. The "12.2% and 3.1% improvements" phrase for HallBench/ChartQA (Section 5.2) reconciles for HallBench as a relative-percent gain (65.0 → 72.9), but ChartQA (83.5 → 87.3) is +4.55% relative or +3.8 pp absolute — neither is 3.1%. The manuscript mixes absolute-pp and relative-percent conventions throughout Section 5.2 without stating which.
• Undefined benchmark (MME-Real) inflates Iteration and Ablation tables. Section 5.1 states seven benchmarks; Tables 2 and 3 include an eighth column "MME-Real" that is never defined, so the "Avg." columns in Table 2/3 are not comparable to Table 1's Avg.
• Table 4 duplicate row label. Two rows in Table 4 are both labelled "InternVL2.5-8B" but with different baseline numbers; the second matches Table 1's InternVL-3-8B row. Almost certainly a mislabel.
• Missing comparison to closely related self-evolving VLM baselines. Vision-Zero and ViPER are positioned as the closest prior work but do not appear in Table 1 or Table 4. The paper's differential claim — that tool-grounded self-evolution outperforms other self-evolving approaches — is therefore not directly evidenced.
• Case studies undermine the tool-grounding thesis. In Figure 8 (geometry) the Verifier's tuples carry tool_check: false and the Q2→Q4 correction is done by re-interpreting an ambiguous instruction linguistically; the tool then only computes lengths inside an already-fixed quadrant. In Figure 10 the tool correctly returns 32.45 but the boxed answer is 32.54. In Figure 12 the code computes 7.7 but the shown sandbox_output is 32.45. In Figure 11 the Verifier's "symmetry violation" critique is mathematically incorrect (roots at $-1, 2$ are consistent with parabolic symmetry about $x = 0.5$).
• No test-time compute analysis. SERC-style inference runs Solver → Verifier (with tool calls per step) → optional repair-conditioned Solver. Comparisons against non-TIR baselines in Table 1 are not compute-matched.
• No calibration analysis of the Verifier. Self-repair gating depends on $\text{conf}_t$ being interpretable, but the paper never reports Verifier accuracy or calibration on any held-out set.
• No per-seed variance. Every reported number is a single run on 8× H200; the Iter-2 → Iter-3 delta (~2 pp) is within typical VLM-RL noise.
• Iterative gains are decelerating. Per-iteration gains (5.2 → 4.0 → 2.8, relative %) approximate geometric decay; the framing "stable and monotonic improvement" overstates what three iterations show without a discussion of saturation.
• Minor presentation issues. Figure 3 uses "Agent-Zero-VL-7B" instead of "Agent0-VL-7B"; Section 5.5 points to "Appendix 5.5" for the full visualization when the actual location is Appendix E, Figure 8.

Reproducibility & code

• Full training pipeline not released. The Agent0-VL repository README states that code release is "coming soon"; no SFT, warm-up RL, or SERC training scripts are shipped. The method depends on many pieces not fully specified in prose — the SERC inner/outer loop implementation, tool sandbox interface, Verifier scoring pipeline, and the SFT-only and Tool-disabled ablation branches — so end-to-end training cannot be reconstructed from the manuscript alone.
• No Agent0-VL checkpoints. Neither the 7B nor 8B trained checkpoints are distributed, so the headline claims (Table 1, Table 2, Table 3, Table 4, Figures 1/3) cannot be independently re-derived. Even releasing only the SFT-initialized (pre-SERC) checkpoint would substantially cut the reproduction cost.
• SFT and RL corpora not shipped. Appendix D describes a multi-stage construction pipeline (Geometry3K, GeoQA, Mulberry, MM-Eureka, Retool, etc.) with GPT-5 / Qwen2.5-VL-72B teacher generation, embedding-similarity dedup, and 10k manual spot-checks. No trajectory JSON, no filter thresholds, and no spot-check labels are released; a reproducer would additionally need API access to a specific GPT-5 snapshot to regenerate the ~240k trajectories.
• Tool sandbox and Verifier interface not released. The whole "tool-grounded" advantage claim depends on a sandbox that executes Solver-generated Python and on Verifier-side re-invocations of the same tools. The tool list, JSON schema exchanged with the sandbox, and safety/timeout configuration are left implicit; the system prompts (Figures 5–7) imply a schema but do not fully define one.
• BoN / PRM evaluation harness not shipped. Table 4 (Agent0-VL as PRM) requires N-candidate generation per query from each of five backbones, followed by Verifier scoring and selection. None of the candidate-generation prompts, the PRM scoring routine, or the selection code are released, and the exact $N$ is only implied by Figure 3 (BoN=8) rather than stated in Section 5.4.
• Seeds and variance not reported. GRPO RL of VLMs is known to have high seed variance; the paper reports a single number per cell in every table, on 8× H200. Without run variance the ordering in the ablation table and the monotone iteration gains are not distinguishable from noise.

Recommended Changes

Essential

• Rephrase the zero-external-reward claim. Rewrite the abstract and Section 1 contribution list so they describe what supervision is absent at the self-evolving stage rather than at the whole pipeline, and explicitly note both the external-reward RL warm-up (Appendix B) and the GPT-5/Qwen2.5-VL-72B teacher distillation for the SFT corpus.
• Add reward-hacking diagnostics. Report at least one quantitative check that the Verifier does not drift or self-inflate rewards over SERC iterations — e.g., Verifier accuracy against an external oracle on a held-out set, correlation between $\text{conf}_t$ and step correctness across Iter 1–3, or the agreement rate between $r_{\text{proc}}$ and $r_{\text{out}}$.
• Restore the four missing display equations for $V_t$, $r_{\text{proc}}^{(t)}$ (Eq. 2), the repair gate $g_t$ (Eq. 3), and the effective step reward (Eq. 4), so the reward function is fully specified.
• Fix the confidence-gate semantics. Split the symbol $\text{conf}_t$ into "Solver step confidence" and "Verifier certainty in critique", state which one the threshold $\tau_c = 0.7$ applies to, and reconcile Algorithm 1's hard threshold with Section 4.2's sigmoid gate (or state that the sigmoid is a soft weighting separate from the hard branching).
• Unify notation. Rename the repair gate away from $g$ to avoid the collision with the trajectory return $g(\tau)$; replace "RA/EE" in Algorithm 1 and Figures 4/8 with the defined $m \in \{S, V\}$; introduce $\eta$ in Eq. 4 (or replace with $C_{\text{repair}}$ in Appendix B); reconcile GRPO group size $N=8$, formal $G$, and "4 rollouts per task".
• Correct the ChartQA figure. Either fix the 3.1% improvement claim to whichever computation (relative % of what averaging window) is intended, or state the convention consistently throughout Section 5.2.
• Define MME-Real or drop it. Describe MME-Real (version, split, protocol) so Tables 2 and 3 are interpretable and comparable with Table 1's seven-benchmark Avg., or remove the column and recompute the averages.
• Fix the Table 4 duplicate row. Change the second "InternVL2.5-8B" row to "InternVL-3-8B" and verify the +Ours delta accordingly.
• Replace or repair the flawed case-study figures. Regenerate Figure 10 so the boxed answer matches the tool output (32.45), regenerate Figure 12 so the shown sandbox_output matches the code result (7.7), remove or correct the "symmetry violation" critique in Figure 11, and either soften Figure 8's caption ("tool-grounded verification") to match the observed tool_check: false behavior or pick a case where a tool call is what surfaces the Solver's error.
• Release the training and evaluation stack. Publish the SFT + warm-up-RL + SERC training scripts, the tool sandbox and JSON schema, the SFT/RL trajectory corpus (or a regeneration script with a fixed teacher snapshot), and at minimum the SFT-initialized checkpoint — the current "Code release coming soon!" state blocks external reproduction of every headline number.

Suggested

• Add a self-evolving-VLM baseline to Table 1 (Vision-Zero and/or ViPER at matched base model), so the claim that tool-grounded self-evolution beats text-only / game-based self-evolution is directly evidenced.
• Report per-seed variance (mean ± std over ≥3 seeds) for at least the main-results and ablation tables, so the Iter-2 → Iter-3 delta and the Self-Repair ablation drop can be interpreted against noise.
• Add a compute-matched comparison against the strongest non-TIR baseline (tokens generated, tool calls invoked, wall-clock), so the gains in Table 1 are not confounded with test-time budget.
• Discuss the observed saturation in Section 5.2 rather than describing three-iteration gains as "stable and monotonic improvement" — an Iter-4 point or an explicit saturation discussion would help.
• Add a Verifier calibration section (accuracy of "high-conf negative-score" flags against an external oracle; ECE of $\text{conf}_t$), which would also support the reward-hacking check above.
• Minor consistency edits. Rename the Figure 3 legend to "Agent0-VL-7B" and fix the "Appendix 5.5" self-reference in Section 5.5 to "Appendix E, Figure 8".