3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models

Summary

3ViewSense targets a real and increasingly discussed gap in modern vision–language models: state-of-the-art LLMs solve olympiad-level symbolic problems yet fail on simple stacked-block counting under occlusion. The paper's diagnostic move is nice — a frozen-feature probe reaches 55.8% where the full VLM is far lower, and a "3-view textual hint" ablation jumps proprietary models by tens of points. From this the authors argue that neither the encoder nor the reasoner is the primary bottleneck; what is missing is a stable, view-consistent intermediate representation between the two. Their proposal, borrowed from engineering drawing, is to make that representation explicit: a set of canonical orthographic projections (front / left / top) that the model first learns to induce (Orthographic Mental Simulation) and then reasons over (View-Grounded Reasoning), followed by GRPO refinement with strict / slack math-verified rewards. To train and evaluate this, they build OrthoMind-3D, a synthetic + sandbox + generative benchmark with a proved uniqueness condition on block heightmaps. Empirically, the framework produces large in-domain gains, non-trivial OOD gains, and consistent transfer to five external benchmarks, all with a 4B base model. The bigger-picture contribution is a legible instantiation of the "structured intermediate representation" idea that improves both accuracy and conciseness — the trained model produces 350-token traces where the base model produces 10k-token drift. The main conceptual limitations sit exactly where the paper hints: three orthographic views are a strong prior that suits stacked cubes and single-plane object layouts but not most open-world scenes; the "3-view hint" experiment risks confounding format with information leakage from the gold structure; and the empirical story rests on a single 4B base model, single-run pass@1 numbers on small OOD splits, and effectively no released data or code. The core idea is worth keeping; the surrounding claims need to be scoped and grounded more carefully.

Strengths

• Conceptual contribution. Framing the VLM spatial-reasoning failure as a missing view-consistent interface, and instantiating it with the classic engineering three-view convention, is a clean, teachable idea that connects diagnostic evidence (probe + hint experiments) to a specific representational choice.
• Diagnostic scaffolding. The two-step diagnosis — probe encoder sufficiency, then test whether an explicit three-view textual context changes accuracy — motivates the architecture before any training is done, which is more informative than the typical "we trained a thing and it works" structure.
• Two-stage decomposition with a clear intermediate. Decoupling view induction (OMS) from view-grounded reasoning (VGR) yields both an ablation-friendly design and an inspectable intermediate; the ordering effect (two-stage $>$ VGR-only $>$ OMS-only in Table 5) is consistent with the framing.
• Response-length and conciseness signal. Table 3 and Figure 13 make a genuinely useful observation: base VLMs generate 10k-token spirals on trivial counting, while the trained model produces short, structured traces — a rare case where an intervention improves accuracy and inference cost together.
• Formal grounding of the data filter. The uniqueness condition (Theorem A.2, Corollary A.3) explicitly connects data curation to a mathematical guarantee about count-label well-posedness, rather than leaving the label validity implicit.
• Transfer breadth. Reporting on five external benchmarks (MindCube-Tiny, CV-Bench 2D, OmniSpatial, SPBench-SI, ViewSpatial) provides more coverage than typical spatial-reasoning papers and the pattern of consistent, if uneven, gains supports the transfer claim at a qualitative level.

Weaknesses

• The "3-view hint" gains may be information leakage, not interface fix. The injected hint (e.g., front-view heights + top-view occupancy) essentially encodes the answer for block counting — top-view occupancy summed against front-view heights is nearly the label. The paper interprets the 30+-point improvements in Figure 1 / Table 11 as evidence that the reasoner is fine and only needs a spatial interface, but no format-matched non-informative control is run to separate "structured format helps" from "gold structural information was pasted in." This is load-bearing evidence for the paper's central hypothesis, so the confound matters.
• The probe result is over-interpreted. 55.8% top-1 on a 50-way count classifier proves that count-relevant signal is decodable from a trainable non-linear head, not that the encoder produces "sufficient geometric information" for the downstream LLM to use. The word "proving" is too strong, and there is no control isolating encoder-side sufficiency from the LLM's ability to actually consume those features under its own conditioning.
• Bijectivity is misstated where it is introduced. The main text says the mapping "from the provided three views to the total cube count" must be bijective, but the count is a single integer and cannot be the target of a bijection over configurations. The mathematically correct statement (configuration $\leftrightarrow$ three projections) appears one sentence later; the earlier phrasing garbles the domain and codomain of the bijection.
• The uniqueness proof has real gaps. In the sufficiency direction, the case-split ($H_{x,y}=1$ with a view-max of 1; or $H_{x,y}>1$ with $H_{x,y}$ strictly larger than $O^c$ or $O^r$) is not obviously exhaustive because Eq.(7) itself is not rendered in the passage. In the necessity direction, the "adjustments can be made" step glosses over the non-trivial bookkeeping needed to preserve all three projections simultaneously when moving mass across cells. Since this theorem underwrites the ground-truth of every In-Domain block-count label, both gaps should be closed.
• Ablations do not isolate orthographic views from generic structured CoT. The Direct-QA ablation removes CoT and answer format at once, so its collapse on SPBench-SI (1.3) and ViewSpatial (7.2) may reflect the model no longer emitting a $\boxed{}$ answer at all, not a failure of view-grounded reasoning. A "free-form CoT with the same teacher" baseline is needed to show that orthographic-view CoT specifically drives transfer, versus any structured CoT.
• OMS-only evaluation is not well-defined. Stage I is supervised only to emit view-JSON, but Table 5 evaluates OMS-only accuracy on tasks with answer targets. The prompt/decoding protocol that elicits an answer from a checkpoint never trained to answer is not described; a "limited performance" finding on this checkpoint could reflect the training objective, not induction quality.
• No error-propagation study on the induced views. The pipeline is critically dependent on $\hat{V}$ being correct, yet the paper never reports OMS output quality against ground-truth projections, nor conditions VGR accuracy on OMS-correct vs OMS-incorrect samples. Given the limitation section acknowledges that three views do not always suffice, this analysis would be more informative than another headline number.
• Single-run pass@1, small OOD sets, no confidence intervals. OOD Object Count / Position sets are 108–109 instances; OmniSpatial Egocentric is 102; SPBench-SI is 306. Several "gains" the paper features (two-stage 34.9 vs VGR-only 32.4 on MindCube; 34.4 vs 34.1 on ViewSpatial) are inside a plausible $\pm 3$–$5$ point 95% CI. Without per-seed variance and Wilson intervals, the reader cannot tell which ranking claims are statistically real.
• Single base model. The "framework" framing rests on a single 4B model (Qwen3-VL-4B-Instruct). The Table 11 hint experiment already shows heterogeneous behaviour across models — e.g., GLM4.1V-9B regresses on block counting with a hint — so the recipe's generality across bases is unsettled.
• Table 2 arithmetic is not internally consistent. The 3ViewSense-4B-sft OOD Block Count reads "1.1 ($\uparrow$46.7%)" and a later 3ViewSense-4B-rl-slack cell reads "50.9 (0.0%)"; neither reconciles with the tabulated Qwen3-VL-4B-Instruct base row. Even if these are typographic accidents, they appear in the most important comparison table.
• Table 11 baseline aggregation is opaque. "Object Reasoning (Overall)" baselines in Table 11 (e.g., Gemini-3-pro 87.4, GPT-5 68.2) cannot be derived from any obvious aggregation of the four Table-1 sub-tasks, and the paper never states how "Overall" is computed.
• Worked examples used to illustrate the method are internally inconsistent. The Block-Counting teaching demonstration (Figure 10) claims 2-high front and left views but reconstructs six blocks all at the bottom level — physically inconsistent. The featured 3ViewSense trace in Figure 13 describes "a stack of two on the left and a stack of three in the center" from the left view even though those two stacks share depth and would overlap in a genuine left projection. The SPBench-SI example in Figure 14 reasons about a "black couch" while the question asks about a pillow. These are the flagship illustrations of "faithful view-grounded reasoning" and they undermine the qualitative claim.
• Reward-curve figure over-reads for downstream accuracy. The claim that OMS-initialised RL causes "degraded downstream performance" cannot come from Figure 5 alone (which plots reward, not held-out accuracy); and the figure crosses initialisation with reward type but the discussion treats it as a one-factor comparison. The label "cumulative reward" is inconsistent with the described oscillation.
• Minor formal issues. The GRPO advantage stabiliser $\delta$ is never assigned a value (and matters when whole rollout groups share a reward); the argument order of $R_{\text{count}}$ and $R_{\text{dir}}$ is inconsistent within the same paragraph; the counting-reward formula is typeset degenerately (a stray hat over the coefficient, and $|y-y|$ instead of $|y-\hat{y}|$). None are fatal but they obstruct careful reading.

Reproducibility & code

The linked repository (https://github.com/Jasaxion/3ViewSense) is effectively empty at review time (README and LICENSE only). The paper claims OrthoMind-3D, the 3ViewSense framework, and the strong empirical gains as its three contributions; none of the artefacts needed to independently check any of them are shipped.

• Missing training and evaluation code. No LLaMA-Factory or verl configs, no OMS/VGR data-construction scripts, no GRPO reward implementation, no evaluation harness for OrthoMind-3D or the five external benchmarks. Table 6 lists high-level hyperparameters but omits seeds, checkpoint-selection criteria, per-benchmark prompt templates for baselines, and decoding parameters.
• Missing OrthoMind-3D dataset. Neither the 19.5k Stage-I set, the 21k Stage-II set, the 30k RL set, nor the ID / OOD evaluation splits (500 + 2202 + 300 + 300 + 500 + 500 ID; 235 + 235 + 108 + 109 OOD) are available for inspection. All headline numbers in Tables 1, 2, 3, 4, 5, 10, 11 rest on this dataset.
• Missing 3ViewSense checkpoints. None of the 3ViewSense-4B-sft, 3ViewSense-4B-rl-strict, or 3ViewSense-4B-rl-slack weights are released; reproducing any row therefore requires re-executing OMS-SFT, VGR-SFT (with a Gemini-3-Flash teacher whose traces cannot be regenerated identically), and GRPO end-to-end from scratch.
• Under-specified probe recipe. The Appendix C.3 probing experiment is the most self-contained result but omits which encoder feature is extracted from Qwen3-VL-4B-Instruct (pooled tokens? projector output?), and the probe's optimiser / lr / epochs / batch size / seed. Even without the OrthoMind-3D data release, this recipe could be fully documented.
• Under-specified baseline evaluation. Proprietary API model IDs, temperatures, system prompts, and the exact inference prompt used on OrthoMind-3D for both proprietary and open-source baselines are absent. This affects Tables 1, 2, and 11.
• Under-specified response-length measurement. Table 3's 10,218.9-token mean is striking but the tokenizer and max_new_tokens are not stated; both materially affect the number.
• Under-specified prompt vocabularies. The Stage-I OMS candidate object list (pyramids + cones) is inconsistent with the 3-view description example (cuboids + cylinders + spheres + cone) in the same figure, and the block-list x axis convention (larger x = further left) may conflict with the "columns from the left" iteration used in narrative traces. Both are concrete obstacles for anyone re-implementing the training pipeline.

Nothing in the release contradicts the paper's claims — but nothing verifies them either. On the veritas coverage axis this is essentially a paper-only submission with a placeholder repository.

Recommended Changes

Essential

• Release the OrthoMind-3D dataset and 3ViewSense checkpoints. Without at least the ID / OOD evaluation JSONLs and one representative checkpoint (RL-slack is most useful given it carries the best OOD numbers), none of the headline claims are independently checkable. Corresponds to the reproducibility weaknesses on missing data and checkpoints.
• Add a format-matched, non-informative control for the 3-view hint experiment. Provide the same textual scaffold with permuted or partially-masked view descriptions to disentangle "structured interface helps" from "gold structural information was injected." This addresses the "3-view hint may be leakage" weakness in Weaknesses.
• Close the uniqueness-proof gaps in Appendix A.1. Render Eq.(7) inline so the sufficiency case-split can be checked for exhaustiveness (including $H_{x,y}=1$ with $M^c_x>1$ and $M^r_y>1$), and add the bookkeeping that shows the necessity construction preserves all three projections when a compensating cell is increased. Addresses the "uniqueness proof has real gaps" weakness.
• Rewrite the bijectivity sentence in Section 3.2 so the object of the bijection is the (configuration, projections) pair, not the count. Addresses the "bijectivity misstated" weakness.
• Report per-seed variance and confidence intervals on small OOD splits. Rerun the RL variants with $\geq 3$ seeds and add Wilson / Clopper–Pearson intervals for OOD Block/Object Count/Position, OmniSpatial-Egocentric (102), and SPBench-SI (306). This addresses the "single-run pass@1" weakness and would sharpen the OOD story.
• Add an OMS-fidelity and error-propagation analysis. Report accuracy of the induced views against ground-truth projections, plus VGR accuracy conditioned on OMS-correct vs OMS-incorrect samples. Addresses the "no error-propagation study" weakness.

Suggested

• Add a "structured CoT without orthographic views" ablation. A teacher-generated CoT baseline that solves the same tasks without the front/left/top scaffold would isolate the value of orthographic views over generic CoT. Addresses the "does not isolate orthographic views from generic CoT" weakness.
• Specify the OMS-only evaluation protocol. State the exact prompt/decoding used to elicit a $\boxed{}$ answer from $M^{\text{SFT}}_{\text{stage1}}$, or drop the row and add an OMS-emitted-view + fixed-extractor variant so the comparison is meaningful. Addresses the "OMS-only evaluation is not well-defined" weakness.
• Rerun the pipeline on at least one additional open-source base (e.g., Qwen3-VL-8B and one of InternVL3.5-8B / GLM4.1V-9B). Addresses the "single base model" weakness.
• Fix the arithmetic and aggregation inconsistencies in Tables 2 and 11. Re-verify every displayed accuracy against its $(\uparrow / \downarrow \%)$ annotation, and state explicitly how "Object Reasoning (Overall)" in Table 11 is aggregated. Addresses the "Table 2 arithmetic" and "Table 11 baseline aggregation" weaknesses.
• Replace the flawed worked examples. Use exemplar figures where the front / left / top view descriptions are geometrically consistent with the final reconstruction (Figures 10, 13) and where the reasoning entity matches the question subject (Figure 14). Addresses the "worked examples are internally inconsistent" weakness.
• Rewrite Figure 5 discussion to (i) label the y-axis as batch/window mean reward rather than "cumulative reward", (ii) discuss the slack-vs-strict axis or drop it, and (iii) either remove the "degraded downstream performance" claim or back it with held-out accuracy from OMS-initialised RL runs. Addresses the "reward-curve figure over-reads" weakness.
• Document the probe recipe and baseline inference configuration in Appendix C.3 and C.2 (encoder feature choice, optimiser, seed, decoding parameters, proprietary API IDs, exact prompts). Addresses the reproducibility notes on probe and baseline evaluation.
• Fix small formal issues. Assign a numeric value to $\delta$ in the GRPO advantage; make $R_{\text{count}}$ and $R_{\text{dir}}$ share an argument order; re-typeset $R_{\text{count}}$ as $\max\{0, 1 - 0.2|y - \hat{y}|\}$; align the front/left/top ordering in $V = \{v_{\text{front}}, v_{\text{left}}, v_{\text{top}}\}$'s parenthetical gloss; align the "attribute-specific vs single-attribute" wording with Table 1's cardinality-vs-attribute contrast. Addresses the "minor formal issues" weakness.