Referee Report — Are VLMs Seeing or Just Saying? Uncovering the Illusion of Visual Re-examination

Summary

The paper asks a simple, well-posed question: when a Vision-Language Model produces a self-reflective phrase such as "let me check the figure again", does it actually re-attend to the pixels, or is the phrase a learned surface pattern? The conceptual move is to answer this behaviorally, via VISUALSWAP — a two-stage protocol that lets the model reason from an image $I_a$, then transparently substitutes it with a visually similar but answer-critical variant $I_b$, and measures whether the model updates its answer. This translates a notoriously slippery introspective question ("is the model really looking?") into a controlled accuracy delta, and the accompanying VS-BENCH (800 curated pairs from MathVista, MathVerse, MathVision, MMMU-Pro) is a nontrivial engineering contribution. The empirical picture is striking and coherent: across 15 open-source VLMs, probe accuracy drops by tens of points; thinking-tuned variants suffer roughly $3\times$ more than instruction-tuned ones; scale offers no protection; an external multi-turn instruction with identical semantic content largely restores accuracy; and an attention analysis plus a $2\times$ image-token amplification intervention give a plausible mechanistic story rooted in insufficient autonomous visual attention. The findings are consequential — they identify a specific failure mode of chain-of-thought "self-verification" in the multimodal regime and reframe it as a control problem rather than a capability problem. The main conceptual limitation is that the framework is deliberately built on visually similar synthetic-vs-real pairs sourced from math/exam-style benchmarks; the paper's most general claims ("say vs. see", generic CoT limitation) outrun this substrate. The mechanistic argument also leans on a single blunt intervention on a single model scale, which is presented as "direct causal evidence" but is more properly evidence that is consistent with — not exclusive to — the attention hypothesis. Several presentation issues (contradictory prompt-robustness statistics, three different layer indices for the same 235B analysis, and blanket phrasings like "strictly monotonic" and "universal recovery" that the tables partly contradict) also need to be tightened.

Strengths

• Clean diagnostic design. VISUALSWAP turns an introspective question ("is the model really looking?") into a measurable accuracy delta by transparently swapping $I_a$ for $I_b$ at the reflection point, with the full re-prefill described token-by-token in Appx. F. The protocol admits both Probe and Multi-turn variants that share content and differ only in turn boundaries — a clean isolation of the mechanism under test.
• Serious benchmark construction. VS-BENCH consists of 800 pairs balanced across four sources with reported CLIP/SSIM/LPIPS metrics and identical resolutions to avoid token-count confounds; a human study confirms the visual differences are detectable; and the annotators are asked to explicitly identify answer-critical elements before generation.
• Broad model coverage and the right controls. Fifteen models across Qwen3-VL, Qwen2.5-VL, ERNIE-4.5-VL, Kimi-VL and reasoning-tuned Qwen2.5-VL variants are evaluated, and the natural confound (that $I_b$ is inherently harder than $I_a$) is directly ruled out by matching baselines on both images (Tab. 3, Tab. 11).
• A layered mechanistic story. The paper does not stop at accuracy: it (i) shows Multi-turn recovery restores accuracy, (ii) links this to a substantial increase in visual attention under identical content, (iii) decomposes the Multi-turn effect into semantic and structural parts (Appx. H), and (iv) closes with a training-free $2\times$ attention-amplification intervention that partially reproduces the recovery — a nicely converging chain of evidence.
• Sharp headline finding on Thinking models. The counterintuitive result that reasoning-tuned models are more, not less, vulnerable to the illusion is a genuinely new and important observation, backed by consistent Instruct-vs-Thinking deltas across every family evaluated and by the stratified same-error analysis in Appx. I.

Weaknesses

• The "original" image is synthetic, not real. Because pairs are sourced from benchmarks that provide $(I_b, Q)$ and $I_a$ is then generated by Nano Banana Pro, every $R_a$ the probe experiment builds on is anchored in an AI-generated image. This asymmetry is not called out in the design principles, and its effect on the probe (e.g. generation artifacts making $I_a$ subtly different in ways the model might latch onto) is not tested with a role-reversed control.
• Robustness statistics contradict the accompanying table. Sec. 5.5 claims paraphrase-driven $\sigma \leq 1.1$, but Tab. 6 reports $\pm 3.1$, $\pm 3.8$, $\pm 4.0$, $\pm 4.9$, and then labels the largest of these ($\pm 4.9$) an "exceptionally tight bound". This is an internal contradiction that materially affects how robust the paraphrase invariance is.
• Three different layer specifications for the 235B attention analysis. The methods text uses "49-52 for 235B", Fig. 3's caption uses "54-57", and the quantitative discussion (and Tab. 5) uses "Layer 55". Without a single consistent specification, the headline $\Delta = 1.07$ / $\Delta = 2.21$ contrast cannot be traced back to a specific reported quantity.
• Attention amplification is under-specified and demonstrated only at one scale. The $2\times$ scaling of attention weights on image tokens is presented as a "training-free" intervention, but the paper does not specify whether the scaling is applied to logits or probabilities, whether the softmax is re-normalized, or how it interacts with vLLM's fused kernels. The intervention is also run only on Qwen3-VL-8B, even though the paper's own scale story predicts a larger gain on 235B-Thinking; the "direct causal evidence" framing is stronger than the demonstration warrants.
• "0% context" endpoint of the length ablation coincides with Base. For Qwen3-VL-235B-A22B-Thinking the "0% retained $R_a$" number is exactly 88.8%, i.e. the reported Base($I_b$). If the 0% cell is simply Base and does not include the reflection prompt $P$, then the monotonic-decline claim spans two conditions that differ in more than just retained context. This should be spelled out.
• Overreaching blanket phrasings partly contradicted by the tables. "Almost entirely eliminates the blindness" (holds for 235B-Thinking; less so for 8B-Instruct where roughly half the gap persists), "strictly monotonic decline in almost every case" (MMMU-Pro Instruct rises at the last step for both 8B and 235B), and "recovery is universal across benchmarks" (residual 6–9 point gaps for 8B-Thinking on MathVerse/MathVision) all move faster than the tables.
• Same-error asymmetry is not decisively evidence of anchoring. Because VS-BENCH is explicitly built on visually similar pairs, a high same-error rate is compatible either with textual inertia (as argued) or with an image-independent tendency to make the same perceptual mistake on similar images. The relevant counterfactual (same-error rate under fresh single-turn inference on $I_b$) is not reported.
• Instruct reflection frequencies in Tab. 7 are elicited, not spontaneous. Only the Thinking-model numbers are true "natural" counts; the Instruct-model numbers measure how often an elicitation prompt succeeds. The subsequent claim that the probe operates on "linguistic patterns the models routinely produce themselves" therefore does not apply cleanly to Instruct models.
• In-family LLM-as-judge. Qwen3-VL-235B-A22B-Instruct is used as the judge for both the main evaluation and the unrelated-image detection-rate control. This means Qwen3-VL models under test — including the 235B-Instruct row itself — are graded by an in-family model, without any cross-judge sanity check. The direction and magnitude of any bias is not discussed.
• Scope is narrower than the framing suggests. All four source datasets are math/exam-style visuals with tightly localized, symbolic answer-critical regions. The paper draws general conclusions about VLM self-reflection and CoT, but the substrate is not representative of natural VQA, document/OCR understanding, or dense-scene tasks. A note on scope and ideally one non-math extension would substantially strengthen the "say vs. see" claim.
• Gemini closed-source generalization is confounded. Because the API does not permit assistant-turn insertion, the Gemini Multi-turn condition uses a lossy "thought summary" as $R_a$ and injects it via a user turn — a structurally different intervention than the open-source Multi-turn. The 86.1% recovery may partly reflect that a summarized $R_a$ carries less inertia to overcome, a caveat that is not acknowledged.
• Presentation nits with real consequences. The Tab. 5 caption reuses "recovers performance" from Tab. 4, but the table reports attention, not accuracy. The "$\Delta = 2.21$ is more than double" framing carries over to an 8B comparison ($4.70 / 2.44 \approx 1.93$) where it does not apply. The scaling summary in Sec. 1 drops the "-Thinking" qualifier from model names, which readers can miss.

Reproducibility & code

• Base and Probe pipelines are covered. The released run_inference.py implements the token-level Probe protocol (image re-prefill with $R_a \oplus P$ inside a single assistant turn) and the standard Base condition end-to-end for all 15 evaluated model families. VS-BENCH itself is fetched via download_data.py, and run_evaluation.py recovers Base($I_a$) vs Base($I_b$) with the correct alternative-answer scoring. The core headline drops in Tab. 2 and the $I_a$/$I_b$ comparability check in Tab. 3 could in principle be reproduced from these artifacts (subject to substantial compute for the 235B rows and stability of the in-family judge).
• Multi-turn recovery has no code path. The paper's second headline — that an explicit multi-turn user instruction restores accuracy to near-baseline — depends on a setting for which no function exists in the release. A reproducer would need to close $R_a$ as a completed assistant turn and re-serialize $I_b$ + $U$ as a fresh user turn using the correct per-family chat template. Since Tab. 4, the mechanistic gap in Sec. 5.3, the decomposition in Appx. H, and the unrelated-image control in Appx. J all depend on this setting, the coverage gap propagates.
• Attention analysis (Sec. 5.3 / Table 5 / Figs. 2–3) is not reproducible from the release. Inference is routed through vLLM, which does not expose per-layer, per-head attention weights, and no HuggingFace-based extraction path or aggregation utility is included. The 100-token-window logic around the intervention point, the case-sampling procedure, and the plotting scripts are also absent, so the concrete numbers underlying $\Delta = 1.07$ / $\Delta = 2.21$ (and the analogous 8B pair) cannot be re-derived without reimplementing all of these ingredients.
• Attention amplification (Sec. 5.7 / Tab. 9) is not shipped. The training-free $2\times$ image-token scaling has no code implementation, no specification of pre- vs. post-softmax scaling, no normalization strategy, and no image-token index derivation per model. This is the paper's main causal-intervention argument and is currently paper-only.
• Robustness sweeps and controls are described but not scripted. The 10 reflection-prompt variants (Table 14), the natural-trigger swap (Tab. 8, which requires an undisclosed trigger vocabulary and — for Instruct models — an undisclosed elicitation system prompt), the decomposition study (Appx. H, five variants), the unrelated-image control (Appx. J, undisclosed source pool and undisclosed detection-rate judge prompt), and the same-error/new-error stratification (Appx. I, manual annotation with no rubric or logs) are all describable but not directly runnable from the release.
• Human study and closed-source evaluation are undocumented. The 5-volunteer $\times$ 50-pair study reports 100% agreement with no instructions, no per-pair responses, and no agreement metrics. The Gemini Multi-turn evaluation (Tab. 18) has no API adapter in the repo; its "thought summary as $R_a$" proxy is not documented at the prompt level.
• Sampling-variance claim ($\sigma < 0.02$). The pipeline hardcodes a single seed and offers no CLI flag to sweep seeds; the passage does not specify which Qwen3-VL model(s), which benchmark(s), or what aggregation was used to compute the $< 0.02$ figure, and the judge-side stochasticity is not separated from decoder-side variance.

Recommended Changes

Essential

• Fix the paraphrase-robustness contradiction. Reconcile the "$\sigma \leq 1.1$" claim in Sec. 5.5 with the $\pm 3.1$ to $\pm 4.9$ standard deviations shown in Tab. 6, and either correct the prose or update the table. Do not describe $\pm 4.9$ as "exceptionally tight".
• Standardize the 235B layer specification. Use one layer or one layer range for the 235B attention analysis (currently 49–52, 54–57, and "Layer 55" all appear), and make the reported $\Delta$ values in Tab. 5 and the text traceable to that specification.
• Specify the attention amplification. State whether the $2\times$ scaling is applied to logits or probabilities, how normalization is handled, and how image-token indices are identified per model. Release the intervention code.
• Run the amplification on at least one large Thinking model. The scale story predicts the largest gain on 235B-Thinking, where the failure is most severe; running only 8B is a mismatch with the "direct causal evidence" framing. Either add the 235B intervention or soften the causal claim.
• Clarify the "0% context" endpoint of the length ablation. State whether it includes the reflection prompt $P$ and the swap continuation, or is identical to Base($I_b$); if the latter, redraw or re-label the curve so the two endpoints differ only in retained-context length.
• Ship the Multi-turn code path. Add a runner that closes $R_a$, appends the user turn with $I_b$ and $U$, and reuses the family-specific chat template, so Tab. 4, Sec. 5.3 (Multi-turn traces), Appx. H, and Tab. 18 can be reproduced without reinvention.
• Release the attention-analysis instrumentation. Provide a HuggingFace-based path that emits per-layer, per-head attention weights over image tokens, together with the 100-token before/after aggregation and the case-selection procedure, so the mechanistic claim in Sec. 5.3 is checkable.
• Add a cross-judge sanity check. Run a subset of the 800 samples through a non-Qwen judge (e.g. GPT-4o-mini or a Gemini/Claude judge) and report agreement or delta on the main Tab. 2 numbers and the unrelated-image detection rates, addressing the in-family evaluation concern.

Suggested

• Add a role-reversed control on a subset. Reverse $I_a$/$I_b$ (probe from real image, swap to the synthetic one) on a small subset to show the effect is not sensitive to which side is synthetic.
• Report a counterfactual for the same-error asymmetry. Compare the Case-2 (same-error) rate to the same-error rate that would arise from independent single-turn inference on $I_b$ for the same subset, so anchoring can be separated from an image-similarity confound.
• Qualify the "universal", "strictly monotonic", and "almost entirely eliminates" phrasings. Tie each of these to the specific model/benchmark cells that support them (e.g., largest Thinking model, non-MMMU-Pro benchmarks) and note the exceptions visible in Tabs. 12 and 13.
• Disambiguate the "up to 60%" vs "20% to 55%" ranges. Update the range in Sec. 5.1 to 20%–60% or explicitly note which models the 55% upper bound refers to.
• Distinguish elicited from spontaneous reflection. In Tab. 7, report unprompted reflection frequency for Instruct models (which may be near zero) alongside the elicited frequency, and restrict the "in-distribution probing" argument to models that actually reflect spontaneously.
• Disclose the natural-trigger vocabulary, the Instruct elicitation prompt, the detection-rate judge prompt, and the unrelated-image source pool. All four are load-bearing for downstream numbers and are currently unspecified.
• Add a small non-math VQA extension. Even a 100-pair subset from natural-image VQA or document-OCR would substantially widen the scope of the "say vs. see" conclusion beyond stylized exam-style visuals.
• Clarify the sampling-variance experiment. State which model, which benchmark, and which aggregation gave $\sigma < 0.02$, and expose a seed CLI flag so the check can be re-run.
• Fix presentation nits. Rewrite the Tab. 5 caption so it describes attention rather than accuracy; either update the "more than double" phrasing or drop the parallel to the 8B pair (whose ratio is 1.93); and add the "-Thinking" qualifier to the scaling comparison in Sec. 1 to match Sec. 4.2.
• Acknowledge the Gemini confound. Note that summarized $R_a$ + user-turn injection is a weaker inertia condition than the open-source Multi-turn setting, so the 86.1% recovery is the upper end of what an API-level analog can show.
• Document the human study. Release the instructions, per-pair responses, and agreement metrics, and consider expanding the sample beyond 5 annotators / 250 pairs to escape the 100% ceiling.