Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning

Summary

The paper reframes the perception-vs-reasoning trade-off in Vision-Language Models as a modality credit-assignment problem: when a VLM answers wrong, was the failure "bad seeing" or "bad thinking"? The method forces the model to externalize perception into <recognition> blocks interleaved with <think> blocks via an instruction prompt, so perception becomes a text-space intermediate that can be independently supervised. Two novel rewards are introduced: Perception Verification (PV) feeds only the perception text to a strong text-only reasoner and rewards the VLM if this "blindfolded" reasoner can still solve the question; Structured Verbal Verification (SVV) asks an LLM judge to execute a step-by-step verification protocol instead of freely judging equivalence, reducing outcome-reward variance. These signals are combined by a Modality-Aware Credit Assignment (MoCA) rule that protects perception tokens on outcome-failure/perception-success trajectories and penalizes them on outcome-failure/perception-failure ones. A single 7B model reports simultaneous gains over its Qwen2.5-VL-Instruct base on nine benchmarks, with the largest jumps on V* (71.4 → 86.6) and MMLongBench-Doc (21.2 → 33.1).

The conceptual move — treating perception as a first-class, separately-supervised action space and reducing "was this perception correct?" to a functional sufficiency test — is worth taking seriously, and SVV is a nice reframing of LLM-as-judge. The chief limitations are: (1) §2.5's central update rule is under-specified — the trajectory return, base advantage, and both MoCA modified advantages are announced with colons but the equations are missing, and the written base advantage uses only $R_O$ despite a summed-return statement; (2) verbal claims outrun the evidence — "precisely routes blame" alongside acknowledged proxy fallibility, "sufficient statistic" from a single fallible proxy pass, and "surpass GPT-4o on reasoning-intensive tasks" although MoCA loses to GPT-4o on MMMU (54.8 vs 63.4) and EMMA (31.3 vs 32.7), with MMMU essentially flat vs base (+0.5); and (3) internal inconsistencies (MoCA/DCA/VL-Scaler naming, an SVV Type D branch writing "Type C Score", DUDE/SlideVQA swap, AVG mis-placed under "Reasoning-Centric," swapped HRBench/InfoVQA drop labels) hurt reproducibility and trust.

Strengths

• Conceptual contribution. The reframing of "seesaw between perception and reasoning" as a credit-assignment problem, and the operational move of externalizing perception into a text intermediate that can be supervised, is genuinely novel and well-motivated relative to the agentic-tools literature that the paper positions against.
• Blindfolded-reasoner sufficiency test. Using a strong text-only reasoner to score "can the answer be recovered from the perception text alone?" is an elegant proxy for the otherwise intractable "is this perception step correct?" question, and it gracefully avoids needing per-question perception ground truth.
• SVV as protocol execution. Turning an LLM judge into an executor of a fixed protocol (identify type → extract → reconstruct → compare) is a sensible way to trade off Rigid Rule brittleness against LLM-Prompting variance; Appendix D disclosing the full VP prompt is a welcome level of detail.
• Broad benchmark coverage. Evaluating a single trained model on nine benchmarks organized into three defensible categories (perception / rich modalities / reasoning) is stronger than the single-benchmark stories common in this space.
• Reliability study on PV. The N=979 human-annotation study against the PV oracle ($\kappa = 0.707$; conservative-FN failure mode) is exactly the right check to do, and the observation that the oracle is biased toward false negatives is genuinely useful because it motivates the "protect" branch of MoCA.

Weaknesses

• Missing equations in §2.5. The trajectory return, the base advantage $A_{\tau,t}$, and both the Case 1 (protect) and Case 2 (punish) modified advantages $A^{MoCA}_{\tau,t}$ are each announced with a colon but the actual equations do not appear. The paper's central update rule is therefore not fully specified.
• Advantage-formula inconsistency. §2.5 opens by saying the trajectory return is the sum of $R_O$ and $R_P$, but immediately defines the group-relative advantage as $R_O(\tau)$ normalized against the group mean. Whether $R_P$ is additive in the return or purely a routing gate cannot be resolved from the text.
• Reward Definition is prose-only. The formal PV reward is announced but not defined; how the 800-token cap interacts with $R_P \in \{0, 1\}$ (violation zeros it? subtracts a fixed amount? applies outside $R_P$?) is left implicit, which matters for reproducibility.
• "Precisely routes blame" overclaim. The paper simultaneously calls the routing "precise" and defends the protect mechanism as necessary because $R_P$ is noisy. Only one of these can be true; the correct claim is probabilistic improvement.
• "Sufficient statistic" overstates a single proxy pass. A binary success of one fallible text reasoner shows adequacy for that reasoner on that question, not the formal sufficient-statistic property, and does not touch the minimality half of the IB objective (a verbose text dump would also pass).
• Information Bottleneck framing is a slogan. The blindfolded reward pushes only on sufficiency; minimality is imposed via a token-length cap, which does not encode $I(V; A_P)$. Calling PV a "functional proxy for the IB principle" is stronger than the mechanism warrants.
• Blindfolded-reasoner premise unsupported. The claim that VL tasks, once visual facts are extracted, are "less complex than pure, abstract reasoning" is load-bearing (it is what makes the 14B text reasoner an "oracle") but stated without evidence or citation, and it is not obviously true on MathVista, EMMA, or chart-based multi-step math.
• Case 2 attributes joint failure entirely to perception. When $R_O = 0$ and $R_P = 0$, perception and reasoning can both be broken. The mechanism amplifies penalty on perception tokens categorically; the hedged word "likely" carries all the weight.
• The $R_O = 1$ branch is not covered. MoCA modifies advantages only for failed trajectories. The "correct answer despite bad perception" case ($R_O = 1$, $R_P = 0$) is left to standard GRPO, which will positively reinforce the flawed perception tokens — undermining the credit-routing story on the positive side.
• "Deterministic" reward signals is inaccurate. Both PV and SVV are LLM-based; the paper itself measures SVV consistency at 92.3% (not 100%) under $T = 0.7$. The correct claim is low-variance/structured, not deterministic.
• "Retains the robustness of agentic workflows" overclaim. Agentic robustness typically comes from multi-turn re-querying with tool feedback; the single-pass autoregressive generation with an internalized "perception function" does not obviously inherit that property.
• Overclaim on GPT-4o. Table 1 does not support "surpasses GPT-4o on reasoning-intensive tasks": MoCA loses on MMMU (54.8 vs 63.4) and EMMA (31.3 vs 32.7); only MathVista is a win.
• Reasoning-centric gains are uneven. MMMU 54.3 → 54.8 (+0.5) is essentially flat, so "massive, broad-spectrum improvements" is not quite accurate. Multiple 7B baselines (e.g., Pixel Reasoner 65.3, VL-Rethinker 56.7) actually beat MoCA on MMMU.
• "58.3 on DUDE" is a typo. Table 1 gives DUDE = 45.1 and SlideVQA = 58.3; the "new standard in rich-modality tasks (58.3 on DUDE)" sentence is mis-labeled.
• HRBench/InfoVQA drop labels swapped in §3.2. Full minus w/o PV gives HRBench $\Delta = 4.1$, InfoVQA $\Delta = 4.7$, but the prose reports them the other way around.
• AVG column mis-placed under "Reasoning-Centric" in Table 5. The reported AVG is a grand average over all nine benchmarks, not a reasoning-centric mean.
• SVV protocol writes "Type C Score" inside the Type D branch. Since SVV's low-variance claim rests on the judge executing a precisely specified protocol, an internal label mismatch in Appendix D step 3.4 is a substantive bug, not a typo.
• Symbol collision on $\beta$. $\beta$ denotes the IB trade-off coefficient in §2.3 and the GRPO KL coefficient in Appendix A.
• $\alpha_{protect} = \alpha_{punish} = 0.2$ with unspecified functional form. Because the §2.5 equations are missing, a single symmetric value can only be interpreted with the equations in hand; no ablation over these coefficients is provided.
• Consistency comparison for Rigid Rule is trivially 100%. Deterministic rules are inherently stable under $T = 0.7$ sampling; presenting 100.0 next to SVV's 92.3 invites the wrong comparison and should be marked "n/a" or dropped.
• Instruction-Only degrades below the base. Table 2's Instruction-Only row is worse than the raw base across the board, which implies part of MoCA's headline gains is the RL run recovering ground the decomposition template gave up. A direct comparison against RL-with-outcome-only on the free-form (undecomposed) CoT would be needed to establish that decomposition + PV beats standard RL.
• Only one base model, one seed, one run per configuration. No variance is reported for Table 1 or Table 2. Some ablation deltas are within 1–2 points (rich-modality; MMMU). VL-Scaler-MiMO in the appendix does not fill this gap because it changes the base.
• VL-Scaler-MiMO contradicts the "fair comparison" clause. §3 states all open-source methods are built on Qwen2.5-VL 7B-Instruct, but Appendix Table 5 adds a variant built on MiMO-VL-Instruct without giving other baselines the same treatment.
• AMT for expert-adjacent judgment without IAA. The PV validation asks AMT annotators to judge "sufficiency" of perception text for benchmarks that span STEM QA and long-form documents; no inter-annotator agreement, qualification, or adjudication protocol is reported.
• Inconsistent naming. MoCA vs DCA (same mechanism), SVV vs VP (same verifier), MoCA vs VL-Scaler (same model in Table 1 vs Table 5). This is not merely cosmetic — it makes the reader wonder whether distinct systems are being compared.

Reproducibility & code

The code and dataset are promised on acceptance; at review time no runnable artifact accompanies the paper. Assessed as read-only paper-code:

• No code, no checkpoints, no data at review time. The novelty is a multi-component training recipe (decomposition prompt, PV blindfolded reasoner, SVV protocol judge, MoCA credit rule, GRPO wrapper), and paper-only reconstruction is blocked by the missing §2.5 equations, undisclosed prompts, and unshared training corpus.
• PV blindfolded-reasoner prompt not shown. The paper specifies the model used for PV (Qwen2.5-Instruct-14B) but not the system prompt — whether the reasoner is told it is blindfolded, how the answer is extracted, and whether success is scored via SVV or a stricter rule.
• LLM-Judge baseline prompt not shown. The w/o SVV+PV (LLM Judge) ablation compares SVV against an unnamed LLM-Judge prompt; without that prompt the "reward-hacking" narrative in §3.2 cannot be interpreted.
• Modality-rich training corpus not released and only aggregate-level described. Table 6 gives source domain and count, but not the crawling → QA-generation → verification pipeline or license terms for the arXiv/newspaper/infographic images. The largest rich-modality gains (e.g., MMLongBench-Doc +11.9) depend on this corpus.
• Per-benchmark evaluation harness / splits / prompts not specified. V*, HRBench (4K vs 8K), InfoVQA (val vs test), MMMU (val vs dev vs test-mini), EMMA (mini vs full), MathVista (testmini vs test), DUDE, SlideVQA, MMLongBench-Doc — none of these are pinned down in the text.
• Single-run, no seeds. Appendix A specifies hyperparameters but no seeds, and Tables 1–2 report single-run numbers; ablation deltas below 2 points cannot be interpreted with confidence.
• VP-Challenge-Set (N=273) not released, and drawn from both train and test. The 273 pairs used to validate SVV are curated from "both the training set and test set," so a reader cannot audit potential test-set touch, and cannot re-run the accuracy/consistency measurements.
• 979-sample PV human-annotation dataset not released, IAA not reported. The confusion matrix in Table 3 is the sole quantitative check of PV; the underlying triplets, per-annotator labels, and inter-annotator agreement are all missing.
• Baseline numbers' provenance unclear. For each cell in Tables 1 and 5 it is not stated whether the value is re-run under the paper's harness or copied from the original paper/leaderboard; V* and MMMU numbers are known to be prompt-sensitive.
• Naming shift between Table 1 (MoCA) and Table 5 (VL-Scaler). Identical numbers under two names, plus an undocumented VL-Scaler-MiMO variant, create confusion about what is being reported.

Recommended Changes

Essential

• Fill in the missing §2.5 equations. Write out the trajectory-level return, the base advantage $A_{\tau,t}$, and both the Case 1 (protect) and Case 2 (punish) modified advantages $A^{MoCA}_{\tau,t}$ explicitly, showing how $\alpha_{protect}$ and $\alpha_{punish}$ enter and how the token-set $\tau_P$ is identified during training.
• Resolve the return/advantage inconsistency. Either revise the "trajectory-level return = sum of $R_O$ and $R_P$" statement or update the base advantage so that $R_P$ actually enters additively; state explicitly whether the MoCA advantage acts only as a routing gate.
• Fix the SVV protocol bug. In Appendix D step 3.4 under IF Type D, change Write: "Type C Score: [score]" to Write: "Type D Score: [score]" and re-report SVV accuracy / consistency after the fix so readers know the numbers reflect the corrected protocol.
• Correct the GPT-4o comparison claim. Re-word "MoCA-7B can even surpass proprietary models, e.g., perception-and reasoning-intensive tasks over GPT-4o" to reflect that MoCA loses to GPT-4o on MMMU (54.8 vs 63.4) and EMMA (31.3 vs 32.7); only V*, HRBench, InfoVQA, and MathVista are wins.
• Temper the "massive, broad-spectrum" wording on reasoning-centric. MMMU improves by +0.5. Either drop the strong wording or add a sentence explaining why MMMU is essentially flat while PV helps EMMA and MathVista.
• Fix the DUDE / SlideVQA number swap in §3.1 ("(58.3 on DUDE)" → SlideVQA) and the swapped HRBench/InfoVQA drop labels in §3.2 (HRBench $\Delta = -4.1$, InfoVQA $\Delta = -4.7$).
• Move or rename the AVG column in Table 5. Place AVG outside the "Reasoning-Centric" grouping so it is clear the value is a nine-benchmark grand average.
• Unify nomenclature. Pick one of MoCA / DCA and one of SVV / VP; keep the model name consistent between Table 1 and Table 5; either drop or fully methodologically describe VL-Scaler-MiMO and reconcile it with the "all open-source methods on Qwen2.5-VL-7B-Instruct" fair-comparison clause.
• Release the artifacts on submission. Publish the training pipeline (MoCA advantage computation, GRPO wrapper, decomposition prompt), the PV blindfolded-reasoner prompt, the LLM-Judge ablation prompt, the modality-rich data pipeline, a Qwen2.5-VL-based checkpoint, and per-benchmark eval scripts. If release must wait, note explicitly for each headline number whether it is re-runnable from what is disclosed.
• Fix the "sufficient statistic" and "deterministic" and "precisely routes blame" phrasings to reflect what the mechanism actually delivers (adequacy for the proxy, low-variance, probabilistic routing).

Suggested

• Add per-seed variance. Report mean $\pm$ std over 3 seeds for the ablation table, at least on rich-modality and reasoning-centric where deltas are small.
• Ablate $\alpha_{protect}$ and $\alpha_{punish}$ independently — the symmetric 0.2/0.2 default begs the question.
• Ablate the 800-token perception cap and the perception weight $\lambda = 0.3$; the IB-inspired minimality argument stands or falls with these.
• Add a baseline of standard RL on the free-form (undecomposed) CoT so the reader can attribute gains to PV/MoCA rather than to the decomposition template.
• Add IAA (e.g., Fleiss's $\kappa$) and annotator qualification for the N=979 PV validation, and release the annotation triplets with per-annotator labels.
• Release the VP-Challenge-Set (N=273) with a per-item source tag distinguishing training-set items from test-set items, so reviewers can audit potential test-set contact.
• Provide per-benchmark evaluation details (split, decoding, prompt template, extractor) in an appendix so the baseline numbers are auditable.
• Cite the source of each baseline row (in-house re-run vs original paper / leaderboard) in Tables 1 and 5.
• Test the framework on a second base model (a non-Qwen 7B VLM) to demonstrate that the gains reflect the recipe rather than base-model idiosyncrasies; the current VL-Scaler-MiMO row is confounded by using a stronger base.
• Drop or annotate Rigid Rule's 100.0 consistency in Table 4 so the SVV-vs-baselines comparison is on a comparable axis.
• Disambiguate $\beta$ by using a different symbol for the IB trade-off vs the GRPO KL coefficient.
• Address the "correct-outcome, bad-perception" branch. Either extend MoCA to $R_O = 1, R_P = 0$ or explain in text why standard GRPO on that branch is acceptable.