Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence

Summary

The paper introduces Holi-Spatial, a three-stage automated pipeline that converts raw indoor videos into rich 3D supervision — a 3DGS reconstruction with rendered depth, open-vocabulary 2D masks, 3D bounding boxes, instance captions, 3D-grounding pairs, and spatial QA — and uses it to release Holi-Spatial-4M, a ~12K-scene dataset drawn from ScanNet, ScanNet++, and DL3DV-10K. The core conceptual move is to replace hand-annotated 3D scans and per-scene language-alignment training with a compositional stack of recent foundation models (Depth-Anything-V3 → 3DGS with normal/render supervision → Gemini3-Pro class memory → SAM3 open-set masks → 2D-to-3D lift with mesh-guided depth filtering → confidence-band gating with a VLM agent-verify step). This is a timely direction: spatial-intelligence VLMs are bottlenecked by supervision from a handful of scans, and a compositional automatic annotation engine could unlock a data flywheel that begins to close the gap with 2D-language pre-training scale. The pipeline is well-motivated, the ablations in Table 5 isolate the contribution of 3DGS training, confidence filtering, and agent recall, and the fine-tuned Qwen3-VL variants show clear improvements on 3D grounding and spatial-reasoning benchmarks. The main conceptual limitation is that the "automation" is demonstrated entirely on the same three indoor datasets the fine-tuned VLMs are later evaluated on, so the paper does not actually demonstrate the scale-to-web-video property it advertises, and the framework evaluation rests on just 10 hand-annotated scenes per dataset, which is far too small to underwrite 60+ AP-point gains without variance analysis. Several headline numbers are also stated with inconsistent units — an "AP50 by 64%" that is a 64-point absolute gap, a "15% AP50" gain that is 14.48 absolute points, a "7.9% MMSI-Bench" accuracy rise that Table 3 shows as a 1.5-point delta (and appears to be mis-attributed from ViewSpatial), and mask/QA totals that disagree across Abstract, Introduction, and Section 4. Executing the released code confirmed that only the curation stack ships — no metric-scoring code (F1/IoU/AP/P25/R25), no SFT/benchmark-eval harnesses, no baseline invocation scripts, no held-out scene manifest, and only a ~2K-scene HuggingFace subset — so none of Tables 2, 3, 4, or 5 can be reproduced from what is public.

Strengths

• Compositional pipeline is well-motivated. Systematically stringing together Depth-Anything-V3, 3DGS-with-normal-supervision, SAM3, a class-memory-conditioned VLM, and an agent-verify band is a plausible way to move beyond scanner-bound 3D datasets, and the paper does a clear job of naming the failure modes each stage addresses (feed-forward depth outliers → 3DGS refinement; SAM3 boundary/depth-edge artifacts → mesh-guided depth filter + mask erosion; SAM3 fragmentation across occlusions → 3D-IoU merge; low-confidence but visually challenging instances → VLM re-verify).
• Concrete artifact, not just method. Beyond a pipeline, the authors release Holi-Spatial-4M (a ~2K-scene HuggingFace subset now, with a ~12K target) and use it to fine-tune two Qwen3-VL sizes, which is a much more falsifiable contribution than a diagram-only proposal.
• Ablations isolate the two most surprising design choices. Table 5 rows 1 vs 2 make it clear that the huge P25/R25 gains hinge on 3DGS refinement rather than DA3 depth alone; rows 3 vs 4 vs 5 justify the tri-band high/verify/low decision rule and show that the agent-verify step is what prevents the confidence filter from destroying recall. Executing the shipped pipeline qualitatively reproduces this direction: on the substitute-input reproduction, the verify band recalled 58 mid-confidence instances and rejected 4, matching the paper's storyline.
• Task breadth of the resulting supervision. Table 1 highlights that no prior 2D-VLM, 3D-VLM, or 3DGS-based approach jointly outputs Depth, 2D Seg, 3D Det, Grounding, and QA — Holi-Spatial-4M gives the community a single dataset that can drive all of them, which is a real ecosystem contribution if the numbers hold.
• Open-vocabulary granularity is directionally supported. On the released ScanNet++ subset the pipeline produced 149,550 instances across 2,775 open-vocabulary classes over 904 scenes — orders of magnitude denser than the ~50 official ScanNet labels the paper contrasts against, which is qualitatively consistent with the Figure 2 granularity story.

Weaknesses

• Headline numbers conflate absolute AP points with percentages. "boost 3D detection AP50 by 64%" and "a 15% AP50 improvement on ScanNet++" are both stated as % but are 64-point and 14.48-point absolute AP differences, respectively. Read literally they understate the true relative gains by ~10x and ~7x. This is the paper's most-cited framing and needs to be fixed.
• A key headline (7.9% on MMSI-Bench) does not match the paper's own table. Table 3 reports Qwen3-VL-8B at 31.1 and Qwen3-VL-8B + Ours at 32.6, a 1.5-point (~4.8%) improvement on MMSI-Bench. The 7.77-point delta actually corresponds to Qwen3-VL-8B's ViewSpatial column (42.35 → 50.12), so the headline appears mis-attributed.
• Dataset scale figures are internally inconsistent. The mask and QA counts are swapped between the Abstract (1.3M masks / 1.2M QA) and the Introduction (1.2M masks / 1.3M QA), and the QA count itself alternates between 1.2M, 1.25M (Section 4 and Figure 5), and 1.3M within the same paper.
• Framework evaluation is under-powered. All of Table 2 and the qualitative figures rest on 10 manually annotated scenes per dataset (30 scenes total), with no confidence intervals, no per-scene breakdown, and no analysis of sampling variance. Given the magnitude of the reported gaps (e.g. Depth F1 0.89 vs 0.39; AP25 81.06 vs 12.2), this is the most serious methodological weakness.
• Training/evaluation domain overlap. Holi-Spatial-4M is built entirely from ScanNet, ScanNet++, and DL3DV-10K, and both the framework and VLM fine-tuning evaluations use those same three sources. The paper's motivating claim — that this pipeline "unlocks abundant web videos" — is therefore not empirically demonstrated on out-of-distribution scenes, and there is no failure-rate report for stage (i) on dynamic or motion-blurred clips.
• Baselines in Table 2 are not like-for-like. SAM3 and SA2VA are single-image 2D methods; SpatialLM/LLaVA-3D/SceneScript consume explicit 3D input; M3-Spatial/LangSplat target language grounding. Presenting Holi-Spatial as the "sole framework capable of simultaneously" producing all three modalities elides these input/task mismatches, which inflates the perceived gap.
• Metric definition is missing. "Depth F1" is the axis of the headline "+0.5 F1" claim yet its inlier threshold, tolerance, and precision/recall convention are never defined; a reader cannot know what a 0.5 gap means.
• Design hyperparameters are unjustified. The 3D-IoU merge threshold $\tau_{merge}=0.2$ is unusually permissive and never ablated; $\tau_{high}=0.9$ and $\tau_{low}=0.8$ carve a very narrow verify band; no sensitivity analysis is offered for any of the three. A 20% overlap threshold in particular risks fusing genuinely separate objects (two chairs at a table) that then propagate as one instance into the caption and QA.
• Class-memory update has no error-correction path. $M_t = M_{t-1} \cup \text{Extract}(I_t)$ is a monotone union: if Gemini3-Pro emits a bad or over-specific label on an early keyframe, it becomes canonical and is reused thereafter. The per-instance confidence filter does not obviously catch systematic labeling drift.
• OBB vs AABB inconsistency. Section 3 and Figure 4 promise an "Oriented Bounding Box (OBB)", but Algorithm 1 uses a generic BBox(P) primitive with no orientation-fit step, and the released output directories are named output_scannetv2_new_aabb/ / output_scannetppv2_new_aabb/. Either the shipped primitive is axis-aligned (then the paper should say so) or the OBB fit is under-specified.
• Algorithm 1 has several printed defects that survive to the released text. The merge-loop guard is written as Cℓ = ∅ (should be ≠); the canonical box $B^*$ is extracted but never written to the output ($B_g$ is); VLMVerify returns a boolean, contradicting the summary text that describes it as producing an updated confidence $s'_k$ to be re-thresholded against $\tau_{high}$; and the merge threshold is renamed from $\tau_{merge}$ to $\tau_{iou}$ with no binding to the numerical value.
• "Agent recall" text does not match the released prompt. Section 3 promises a VLM "equipped with an image zoom-in tool and a SAM3 re-segmentation tool" whose output is an updated confidence, whereas the Figure 12 template is a single-turn ACCEPT/REJECT classifier that takes three pre-baked images and calls no tools.
• "Outperform human annotations" is overstated. The evidence is a granularity count against one dataset's official labels (Figure 2), yet the very evaluation used to certify the pipeline's quality re-annotates scenes "with human supervision" as GT.
• Table 3 wins are more mixed than the prose says. On MMSI-Bench the 8B and 2B deltas are +1.5 each; Qwen3-VL-2B + Ours (27.6) still trails VST-SFT-3B (30.2) and VST-SFT-7B (32.0) on the very benchmark the headline names.
• Single-backbone downstream evidence. Every downstream result fine-tunes Qwen3-VL (2B, 8B) only, so it is not established that Holi-Spatial-4M is a dataset property rather than a Qwen3-VL property. At least one additional backbone would substantially strengthen the payoff claim.
• Compute/cost of the fully automated pipeline is opaque. Every stage runs an expensive foundation model (Depth-Anything-V3, per-scene 3DGS, Gemini3-Pro, SAM3, Qwen3-VL-30B, agent-verify VLM), yet no GPU-hours, VLM tokens, or wall-clock per scene are reported, which matters directly for the scalability argument.
• Minor presentation issues. Figure 1's caption attributes the +0.5 F1 / +64% gains to ScanNet while the body attributes them to ScanNet++; Figure 9 labels two bottom panels as ID 5 when the second should be ID 4; the Section A.1 sentence "for example, in Section A.1, a complete bed is detected as two separate beds" self-refers to the section the reader is already inside; Figure 15's three-column layout detaches option keys from their answers.

Reproducibility & code

• Repository ships only the curation pipeline, no scoring code. Executing the shipped code end-to-end shows the release contains DA3 depth, PGSR/3DGS training, SAM3 masking, 2D→3D lifting, and the VLM verify wrapper — but no Depth F1, 2D IoU, 3D AP25/AP50, or P25/R25 scoring code. The only IoU/box-ops symbols in the tree are internal SAM3 mask utilities. As a result, none of Table 2, Table 4, or Table 5 can be recomputed from what is public.
• No SFT or benchmark harness for the downstream tables. MMSI-Bench, MindCube, ViewSpatial, SparBench-tiny, and the ScanNet++ 3D-grounding AP15/25/50 eval that anchor Tables 3–4 are all external to the repo, and the LLaMA-Factory SFT conversion step (Step 9 in the reproduction) is blocked upstream. Neither the fine-tune nor the benchmark scoring can be reproduced from what is shipped.
• Fine-tuning recipe is under-specified. The paper reports only "1 epoch", "batch size 1024", and "32 H800 GPUs", but omits learning rate, optimizer, warm-up/schedule, LoRA vs full-parameter tuning, seed, sequence length, dropout, and frozen-vs-trainable modules. The +14.48 AP50 grounding gain and the (already-inconsistent) MMSI-Bench delta therefore have no runnable recipe.
• Raw ScanNet++ imagery and official SAM3 weights are gated. Every generation stage reads the raw ScanNet++ DSLR imagery, which is behind the TUM ScanNet++ license; the SAM3 checkpoint used by default (facebook/sam3) is behind Meta's gated-access flow. The reproduction had to render 32 views from a released Gaussian model as a stand-in and swap in the community mirror 1038lab/sam3 for SAM3. Neither dependency nor the substitution path is documented in the README.
• Dataset release is a subset, not the full 12K. The linked HuggingFace share yielded a ~2K-scene / 904-ScanNet++ subset — an order of magnitude smaller than the 12K figure the Abstract and Section 4 quote — and the 4M+ annotation totals (1.3M/1.2M masks, 320K bboxes, 320K captions, 1.2M grounding, 1.2M QA) cannot be verified from what is released. No manifest links annotations to scenes.
• The 10 held-out GT scenes are not identified. Scene IDs, the class taxonomy used for re-annotation, and the annotation files for the 30 scenes underlying Table 2 and Table 5 are not shipped; even a reproducer with identical checkpoints and a correct scorer cannot re-derive the reported numbers.
• Baselines are not runnable end-to-end. Baselines in Table 2 (M3-Spatial, LangSplat, SA2VA, SpatialLM, LLaVA-3D, SceneScript) have no invocation scripts or shared-scorer glue in the release, so the "0.64 vs 0.25 IoU" and "70.05 vs 6.23 AP50" cells are currently paper-only.
• Release-hygiene defects a reproducer will hit immediately. The .gitignore excludes the sam3/train/ subpackage, so the vendored SAM3 does not import out of the box (the reproduction reconstructed __init__.py, masks_ops.py, matcher.py, data/collator.py as re-export shims). Several runtime dependencies are undeclared (pytorch3d, psutil, ftfy, regex, manifold3d), the vendored 3DGS CUDA code needs a <float.h> include, and the data-download API expected by the pipeline needs a swap to the HuggingFace client.
• Directional consistency, not magnitude. Where mechanism could be checked, the pipeline behaved as the paper describes: the verify band routed mid-confidence instances through the VLM (58 recalled, 4 rejected), open-vocabulary curation over the released subset yielded 2,775 classes vs the ~50 official labels, and box outputs matched the released dataset schema. But no headline number was numerically confirmed, and the discrepancy is driven by missing scoring code plus gated inputs rather than a computed contradiction.

Recommended Changes

Essential

• Reconcile the headline framings with actual metric units. Rewrite "boost 3D detection AP50 by 64%" and "a 15% AP50 improvement" to explicitly say ~64 absolute AP50 points (70.05 vs 6.23) and ~14.5 absolute AP50 points (27.98 vs 13.50). Disambiguate the "7.9% MMSI-Bench" claim against Table 3's 1.5-point delta — either correct the headline (it appears to match ViewSpatial, not MMSI-Bench) or correct the table.
• Unify dataset totals across Abstract, Introduction, Section 4, Figure 5, Conclusion, and Section 5.2. Pick one canonical value for masks (1.2M vs 1.3M) and one for spatial QA (1.2M vs 1.25M vs 1.3M) and enforce it everywhere.
• Release the metric-scoring code and evaluation harnesses. Ship the Depth F1, 2D IoU, 3D AP25/AP50, and P25/R25 scorers used for Tables 2 and 5, plus the MMSI-Bench / MindCube / ViewSpatial / SparBench-tiny / ScanNet++ 3D-grounding wrappers used for Tables 3–4. Without them, the paper's five headline tables are un-checkable.
• Release the pipeline configurations and the exact fine-tuning recipe. Provide the 3DGS training config, the 2D-to-3D lifting/mesh-guided-depth-filter implementation, the tri-band confidence filter with $\tau_{high}=0.9$ and $\tau_{low}=0.8$, the agent-verify wrapper (including any tool-use loop, if it exists), and the SFT recipe — learning rate, optimizer, warm-up/schedule, LoRA-vs-full-parameter, sequence length, seed, and the frozen vs trainable modules.
• Publish the 10-scene held-out list and GT annotations. Give the exact scene IDs (per dataset) and the 2D-mask / 3D-bbox annotation files used for Tables 2 and 5, together with the class taxonomy and the P25/R25 evaluation script.
• Enlarge the framework evaluation set and report variance. Increase beyond 10 scenes per dataset, and report mean±std across multiple random samples for at least the Depth F1, 2D IoU, and AP25/50 headline cells.
• Fix the release-hygiene defects. Audit .gitignore so sam3/train/ ships, declare every runtime dependency (pytorch3d, psutil, ftfy, regex, manifold3d), patch the missing <float.h> include in vendored 3DGS CUDA code, and document the gated-input path for ScanNet++ DSLR imagery and facebook/sam3 weights.
• Fix Algorithm 1 defects. Correct Cℓ = ∅ to Cℓ ≠ ∅; remove or actually use $B^*$; align VLMVerify with the summary text (either return an updated $s'_k$ score or restate the text as a boolean); and add a line binding $\tau_{iou} \equiv \tau_{merge} = 0.2$, $\tau_{high}=0.9$, $\tau_{low}=0.8$.
• Align the agent-verify description with the released prompt. Either implement the promised zoom-in and SAM3 re-segmentation tools with a runnable agent loop that produces an updated $s'_k$, or rewrite Section 3 to describe the single-turn ACCEPT/REJECT classifier that Figure 12 actually is.

Suggested

• Add an out-of-distribution evaluation. Run the pipeline (or at least stages (i)+(ii)+(iii) on a handful of scenes) on raw web videos or a non-ScanNet-family scan, and report failure rates for stage (i) under motion blur, low parallax, and dynamic objects, to substantiate the "scalable to abundant web videos" framing.
• Ablate the hyperparameters that were introduced without justification. Sensitivity of P25/R25 (and downstream Qwen3-VL AP50) to $\tau_{merge}$, $\tau_{high}$, and $\tau_{low}$.
• Add a downstream 3DGS-refinement ablation. Show whether "Holi-Spatial-4M with 3DGS-refined depth" vs "same pipeline with DA3 depth only" changes the Qwen3-VL 3D-grounding AP50 and MMSI-Bench accuracy, so the fine-tuning gains can be attributed to geometric refinement.
• Add a second fine-tuning backbone. At least one non-Qwen3-VL VLM (e.g. InternVL3, LLaVA-1.5) fine-tuned on the same QA/grounding data would confirm that the payoff is a dataset property, not a Qwen3-VL property.
• Add a compute/cost table. GPU-hours per scene for 3DGS training, VLM tokens per scene for Gemini3-Pro / Qwen3-VL-30B / agent-verify, and total wall-clock/monetary cost to produce Holi-Spatial-4M.
• Define Depth F1 precisely. Give the inlier threshold, resolution, and precision/recall convention in the main text and in the released evaluation script.
• Reconcile the OBB vs AABB story. Either fit and evaluate a true oriented bounding box (specifying the orientation-fitting method), or rename the primitive to AABB in the text and figures.
• Describe how erroneous class-memory entries are corrected. Either specify a pruning/merging rule for $M_t$, or ablate the effect of a corrective update on P25/R25.
• Soften the "outperform human annotations" framing to "denser and finer-grained than official ScanNet labels", or add an inter-annotator-agreement study against the human-re-annotated GT.
• Rename artifacts consistently. Use Holi-Spatial (framework) and Holi-Spatial-4M (dataset) throughout, and drop the Abstract's Holi-Spatial, the first ... multimodal dataset phrasing that collides with the later Holi-Spatial-4M, the first largescale, high-quality 3D semantic dataset.
• Soften the Table 3 "consistently improves" framing and comment on why the MMSI-Bench delta is small (+1.5 points) relative to MindCube (+10 to +19).
• Fix minor figure/appendix issues. Correct Figure 1's caption to attribute +0.5 F1 / +64% to ScanNet++; relabel Figure 9's bottom-row Confidence-Filter panel as ID 4; replace the Section A.1 self-reference with a Figure 11 pointer; re-typeset Figure 15 so each QA's options and answer key are unambiguous.