SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models

Summary

SpatioLM targets a real gap: current VLMs handle language and semantics well but consistently underperform on spatial-metric and 3D-reasoning tasks. The authors' conceptual move is to argue that pretrained VLMs already contain latent geometric knowledge, and that it can be elicited rather than either injected via external 3D encoders or overwritten via invasive fine-tuning. Mechanically, the frozen VLM is augmented with a ControlNet-style side path (the Spatio-Vision Module: six 50M-parameter blocks with alternating intra-frame and inter-frame attention), whose outputs are fused back into every $k$-th LM block through zero-initialized projections and simultaneously supervised by a Dual DPT head predicting pseudo depth and camera ray maps generated by Depth Anything V3. This design cleanly decouples "spatial capability" from "backbone drift" and lets the same trained system tackle depth QA, video-based spatial reasoning (VSI-Bench), scene QA (ScanQA, SQA3D), and VLA manipulation with a shared plug-and-play head.

The empirical case is broad: SpatioLM (SenseNovaSI-8B backbone) reports a 71.6 VSI-Bench average, state-of-the-art depth QA, competitive VLA success rates on LIBERO, and only ~3–4% drop on VideoMME/MVBench/MMMU/VideoMMMU versus ~44% for Spatial-MLLM. The ablations show internally consistent, orthogonal contributions from the SV-Module, VTS, and DGS.

The main conceptual limitations are threefold. First, the headline dominance is heavily backbone-dependent — the SenseNovaSI-8B variant beats the InternVL3.5-8B variant by 30+ points on MD-S and 4–5 on VSI-Bench — so it is unclear how much of the gain comes from "elicitation" versus from a backbone already strong at spatial tasks. Second, training and evaluation draw on the same ScanNet-family scene distributions, and the paper does not audit scene-level overlap with VSI-Bench / ScanQA / SQA3D. Third, the "preserve general capabilities" story is more nuanced than framed: MVBench motion subtasks drop by 24–30 points, which the paper honestly reports but the abstract and conclusion do not qualify. Reproducibility is currently blocked by unreleased code, checkpoints, custom evaluation splits, and backbone weights.

Strengths

• Conceptual contribution. The reframing from "inject 3D priors / add encoder" to "elicit latent geometry from intermediate ViT tokens with a frozen backbone" is clean and well motivated. Selecting tokens from the 16th of 24 ViT layers, justified by prior evidence that middle layers carry richer spatial signal, is a specific and defensible design choice.
• Non-invasive side-path design. Freezing the VLM and combining a side path with zero-initialized projections is a genuinely non-invasive integration, and the general-capability comparison against Spatial-MLLM (Fig. 6, Table 14) provides an unusually clear demonstration that catastrophic forgetting can be avoided in practice, not just in principle.
• Coverage across the stack. A single training pipeline supports low-level depth QA (MD-S/MD-M/DR/DA-2K), high-level video-based spatial reasoning (VSI-Bench), 3D scene QA (ScanQA/SQA3D), and embodied manipulation (LIBERO). The VLA transfer (91.0% discrete / 96.3% continuous) without large-scale robot pretraining is a meaningful demonstration of transferability.
• Ablation completeness. Tables 5–8 disentangle the SV-Module, VTS, DGS, attention mechanism, block count, and loss weights; deltas across configurations are arithmetically consistent (e.g., Alt-Attn +1.2 / +3.1 / +1.8; LoRA delta +1.8 / +4.6 / +2.9), which increases trust in the reported ordering.
• Honest limitations section. The Motion-dynamics trade-off and pseudo-label dependence are called out explicitly in the limitations, and the MVBench subtask breakdown in Table 16 exposes exactly where the method regresses.

Weaknesses

• Backbone-dependence of headline numbers. The SenseNovaSI-8B variant reports 83.5/69.0/83.8/41.8 on the depth benchmarks and 71.6 on VSI-Bench, while the InternVL3.5-8B variant reports 50.9/32.0/81.7/24.1 and 65.0. The 30-point MD-S gap makes it hard to separate "elicitation of latent geometry" from "SenseNovaSI-8B was already strong at spatial tasks." The paper should isolate what property of the SenseNovaSI backbone is being elicited (e.g., pretraining exposure).
• Training/evaluation scene overlap not audited. Training uses ScanNet, ScanNet++, ARKitScenes, Hypersim, Waymo (Table 9). VSI-Bench, ScanQA, and SQA3D are built on ScanNet-family scenes; MD-M uses NRGBD/ScanNet/KITTI. Official train/val split separation is not the same as scene-level separation for RGB-D scans; the paper does not report room- or scene-ID overlap, which is a plausible confound behind the 71.6 vs. 60.6 (Qwen3-VL-235B) gap.
• Overstated dilemma. The Related Work casts prior work as a strict "external-dependencies vs. invasive-modification" dichotomy, then positions SpatioLM as the escape. Dataset-centric works (including SenseNovaSI itself) already occupy a middle ground without external inference dependencies or architectural invasiveness. This framing overstates the novelty axis and hides that SpatioLM's own frozen backbone is one of these methods.
• Loss objective underspecified. Three weights $\alpha, \beta, \gamma$ are stated, but the objective has at least four terms (LM, distillation, depth, ray). The full equation is not rendered in the extraction; how $L_{de}$ and $L_{ray}$ are internally balanced under a single geometric weight is not specified.
• Intrinsic-free claim overreaches. The paper argues the pipeline avoids camera intrinsics for "real-world deployment," but the DGS ray-map targets are derived from camera intrinsics/extrinsics of the source datasets. The claim holds for inference and QA-target construction, not for supervision.
• General-capability preservation is uneven. MVBench motion-intensive subtasks drop by 24–30 points on both backbones (Table 16). The abstract's "effectively limiting the degradation of general capabilities" and the conclusion's "effectively mitigates the degradation" do not communicate this trade-off, and the causal attribution ("geometry-oriented supervision emphasizes static spatial structure") is asserted rather than isolated by ablation.
• No seed variance / significance. All results are single-run point estimates. Several key deltas (K=6 vs. K=8 on VSI-Bench: 71.6 vs. 71.7; loss-weight spread: <0.5 points) are within typical single-seed noise, yet none of the ablation conclusions acknowledge the statistical uncertainty.
• Table interpretation issues. The "Avg." columns in Tables 10 and 13 mix $\delta<1.25\uparrow$ (0–100 scale) with Absolute Relative error $\downarrow$ (values spanning three orders of magnitude, e.g., 386, 344, 131) without stating the aggregation. Some claimed superlatives (best on DA-2K) are actually ties (VST-7B-SFT 83.8 vs. SpatioLM 83.8).
• Qualitative claims used as evidence. The "consistent with textual outputs" argument (Fig. 12) and the "<0.3 m error" claim (Fig. 11) are both drawn from qualitative visualizations; Fig. 11 does not include ground-truth values, so the error claim is not verifiable from the figure.
• VLA experiment thin on detail. The +11.8-point discrete improvement over SenseNovaSI-VLA0 is impressive, but the LIBERO training data, action-chunk length, ensemble window, and SV-Module→action-head interface are all missing.
• Minor prose issues. $\hat{R}$ is defined as a per-pixel ray field but later described as "camera parameter values"; the DPT head input is $H^g$ without an index despite six $H_i^g$ streams; the DR section drops Cross-View Distance from the enumeration; the acknowledgments dataset list does not match Table 9; RoboRefer and RefSpatial-Bench share a citation without noting shared authorship.

Reproducibility & code

Coverage note: this is a read-only inspection (paper-code mode, no execution). The released codebase/ directory is empty in the evidence bundle, and the paper contains no explicit code, model, or data-availability statement. The veritas assessments mark every one of the 18 extracted claims as "partial" support with mostly high or medium reproducibility risk, driven by the following gaps:

• No code or model release. The SV-Module implementation, both SpatioLM checkpoints (SenseNovaSI-8B and InternVL3.5-8B backbones), and the VLA variants are not distributed. Every headline number in Tables 1–5 and 12 therefore relies on unreleased artifacts. This should be addressed with a code+checkpoint release or, at minimum, a public research-license commitment.
• Custom benchmarks not shipped. MD-S, MD-M, and DR are constructed by the authors from public source data (SUN RGB-D, NYUv2, Waymo, NRGBD, ScanNet, KITTI, 7-Scenes) with random marker placement and no released seed. Independent construction produces a different evaluation set, so the reported 83.5 / 69.0 / 41.8 cannot be reconstructed even given a trained model.
• Training hyperparameters missing. The peak learning rate, per-GPU batch size, effective global batch size, gradient accumulation, seed, samples per epoch, and mixed-precision setting are not reported. The unusual 600-vs-2 epoch asymmetry between perception and understanding tasks compounds the ambiguity.
• Distillation and DGS targets underspecified. Which Depth Anything V3 variant / layer is used as VTS teacher, how student and teacher features are spatially aligned/projected, and how the pseudo depth and ray targets are generated are not described in enough detail to reproduce DGS or the VTS ablation.
• Backbone availability unclear. SenseNovaSI-8B and SenseNovaSI-2B are cited but the paper does not confirm they are publicly available checkpoints. If they are not, external reproduction of even a frozen forward pass is blocked.
• VLA recipe missing. VLA-0 and OpenVLA-OFT are named, but the LIBERO training corpus, action-chunk length, ensemble window, and SV-Module→action-head interface are not detailed; the 91.0% / 96.3% results are effectively opaque.
• Evaluation protocol not pinned. VSI-Bench prompt template, numeric-answer parser tolerance, ScanQA BLEU/CIDEr scorer implementation, and SQA3D E1/ER1 normalization scripts are not stated. These commonly shift reported numbers by 1–3 points across independent evaluators.

Recommended Changes

Essential

• Release code, checkpoints, and evaluation splits. Ship the SV-Module implementation, both SpatioLM training/eval configs, the two trained checkpoints, and the exact MD-S/MD-M/DR splits (indices, marker overlays, prompt templates, answer parser). This addresses the "no code/checkpoint" and "custom benchmarks not shipped" gaps in Reproducibility & code.
• Add a full hyperparameter and evaluation-protocol table. Report peak LR, per-GPU batch size, effective global batch size, gradient accumulation, seed(s), samples per epoch for each task category, mixed precision, VSI-Bench prompt template, ScanQA/SQA3D scorer versions, and the numeric-answer parser tolerance for depth QA. Cover the "training hyperparameters missing" and "evaluation protocol not pinned" gaps.
• Audit training/evaluation scene-level overlap. Report room- or scene-ID overlap between training scenes (ScanNet, ScanNet++, ARKitScenes, Hypersim) and the ScanNet-family evaluation benchmarks (VSI-Bench, ScanQA, SQA3D, MD-M) and discuss whether the InternVL3.5-vs-SenseNovaSI backbone gap correlates with pretraining exposure. This addresses the "training/evaluation scene overlap not audited" weakness.
• Report seed variance on headline results. Run at least three seeds for the SenseNovaSI-8B configuration on VSI-Bench, MD-S, DA-2K, ScanQA and SQA3D and add standard deviations. This is needed so readers can judge the +1.2 / +1.8 / +0.5 deltas throughout Tables 5–8 and 12 against noise.
• Isolate what is being elicited from the backbone. Add an analysis explaining why SpatioLM-SenseNovaSI-8B outperforms SpatioLM-InternVL3.5-8B by 30+ points on MD-S and 4–5 on VSI-Bench (e.g., ablate initialization from a weaker backbone, or freeze SenseNovaSI and swap in only its vision encoder), addressing the "backbone-dependence" weakness.
• Write the full loss equation and clarify weighting. Make explicit which of $\alpha, \beta, \gamma$ scales the LM, VTS, and DGS terms, and how $L_{de}$ and $L_{ray}$ are combined within DGS. Resolves the "loss objective underspecified" weakness.
• Scope the "no camera intrinsics" claim. Rewrite the training-side and evaluation-side sentences to make clear that intrinsics/extrinsics are used only to build DGS ray-map targets, not at inference or for QA-answer construction. Resolves the "intrinsic-free claim overreaches" weakness.

Suggested

• Qualify the "first to surpass 70" claim. Scope it to the compared open-source/specialized-spatial models, matching what Table 2 actually shows.
• Rewrite the "preserves general capabilities" framing. State up front (abstract and conclusion) that scene-level reasoning is preserved but motion/dynamics subtasks in MVBench drop 24–30 points, matching the honest limitations paragraph.
• Test the causal claim about motion degradation. Replace DGS with a temporal-signal supervision (e.g., optical-flow prediction), or ablate DGS alone, to isolate whether "static-vs-temporal competition" is really the mechanism behind the MVBench pattern.
• Parameter-match the self-attention ablation. Confirm whether the self-attention baseline in Table 7 has the same parameter count / FLOPs as alternating attention; if not, add a matched variant so the +1.2 / +3.1 / +1.8 gains can be attributed to the inductive bias rather than capacity.
• Reconcile "no additional inference overhead" with the ~0.3B-param prefill. Report wall-clock or FLOPs for the SV-Module's prefill pass on a representative 32-frame 448×448 input relative to base VLM prefill.
• Fix taxonomy and notation drift. Group dataset-centric works separately in Related Work; use $H_i^g$ consistently and specify which block's tokens feed the DPT head; unify $\hat{R}$'s description as a dense ray field throughout; enumerate all five DR subtasks and note their input arity; add per-scenario size counts so 590K + ScanQA + SQA3D reconciles to 643K.
• Add ground-truth annotations to Fig. 11 so the "<0.3 m error" claim is visually verifiable, or replace it with a per-scene error table computed over the MD-M split.