SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning

• URL: http://arxiv.org/abs/2603.27437v1
• Date: Sat, 28 Mar 2026 22:49:40 GMT
• Title: SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning
• Abstract summary: Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems.<n>We propose SpatialStack, a hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy.<n>Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks.
• Score: 22.547972947051765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems. This limitation arises from their inability to capture fine-grained 3D geometry and spatial relationships. While recent efforts have introduced multi-view geometry transformers into VLMs, they typically fuse only the deep-layer features from vision and geometry encoders, discarding rich hierarchical signals and creating a fundamental bottleneck for spatial understanding. To overcome this, we propose SpatialStack, a general hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy. Moving beyond conventional late-stage vision-geometry fusion, SpatialStack stacks and synchronizes multi-level geometric features with the language backbone, enabling the model to capture both local geometric precision and global contextual semantics. Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks. Extensive experiments and ablations demonstrate that our multi-level fusion strategy consistently enhances 3D understanding and generalizes robustly across diverse spatial reasoning tasks, establishing SpatialStack as an effective and extensible design paradigm for vision-language-geometry integration in next-generation multimodal physical AI systems.

Related papers

• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models [32.477423778941386]
  We present GeoVR, a framework that learns geometric representations using purely 2D video sequences.<n>Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge.<n>Experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-06-04T08:11:12Z)
• HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation [83.31948299340782]
  HERMES++ is a unified driving world model that integrates 3D scene understanding and future geometry prediction within a single framework.<n>Hermes++ achieves strong performance, outperforming specialist approaches in both future point cloud prediction and 3D scene understanding tasks.
  arXiv  Detail & Related papers  (2026-04-30T17:59:58Z)
• Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images [81.94999489820974]
  UniSplat is a feed-forward framework for learning 3D representations from unposed multi-view images.<n>We introduce a dual-masking strategy that strengthens geometry induction in the encoder.<n>Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies.<n>Third, we introduce a pose-conditioned recalibration mechanism that interrelates predicted 3D point and semantic maps into the image plane.
  arXiv  Detail & Related papers  (2026-04-12T10:36:18Z)
• Let Geometry GUIDE: Layer-wise Unrolling of Geometric Priors in Multimodal LLMs [13.627465963609936]
  GUIDE (Geometric Unrolling Inside MLLM Early-layers) is a progressive geometric priors injection framework.<n>We introduce a context-aware gating that enables the model to fetch requisite spatial cues based on current semantics.
  arXiv  Detail & Related papers  (2026-04-07T10:45:28Z)
• GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models [70.61152292499737]
  We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm.<n>Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision.<n>We propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation.
  arXiv  Detail & Related papers  (2026-03-17T12:43:48Z)
• SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs [21.891285551179365]
  We introduce Spherical Coordinate-based Positional Embedding (SoPE)<n>Our method maps point-cloud token indices into a 3D spherical coordinate space, enabling unified modeling of spatial locations and directional angles.<n>This formulation preserves the inherent geometric structure of point-cloud data, enhances spatial awareness, and yields more consistent and expressive geometric representations for multimodal learning.
  arXiv  Detail & Related papers  (2026-02-26T07:42:15Z)
• Thinking with Geometry: Active Geometry Integration for Spatial Reasoning [68.59084007360615]
  We propose GeoThinker, a framework that shifts paradigm passive fusion to active perception.<n>Instead of feature mixing, GeoThinker enables the model to selectively retrieve geometric evidence conditioned on its internal reasoning demands.<n>Our results indicate that the ability to actively integrate spatial structures is essential for next-generation spatial intelligence.
  arXiv  Detail & Related papers  (2026-02-05T18:59:32Z)
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling [68.14113731953971]
  This paper introduces MILO, an Implicit spatIaL wOrld modeling paradigm that simulates human-like imagination.<n>We show that our approach significantly enhances spatial reasoning capabilities across multiple baselines and benchmarks.
  arXiv  Detail & Related papers  (2025-12-01T16:01:41Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.