SpatialForge: Bootstrapping 3D-Aware Spatial Reasoning from Open-World 2D Images

• URL: http://arxiv.org/abs/2605.11462v1
• Date: Tue, 12 May 2026 03:20:41 GMT
• Title: SpatialForge: Bootstrapping 3D-Aware Spatial Reasoning from Open-World 2D Images
• Abstract summary: We propose a scalable data synthesis pipeline that transforms in-the-wild 2D images into spatial reasoning supervision.<n>Our approach decomposes spatial reasoning into perception and relation, and constructs structured supervision signals covering depth, layout, and viewpoint-dependent reasoning.
• Score: 16.243887520581925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in Large Vision-Language Models (VLMs) have demonstrated exceptional semantic understanding, yet these models consistently struggle with spatial reasoning, often failing at fundamental geometric tasks such as depth ordering and precise coordinate grounding. Recent efforts introduce spatial supervision from scene-centric datasets (e.g., multi-view scans or indoor video), but are constrained by the limited number of underlying scenes. As a result, the scale and diversity of such data remain significantly smaller than those of web-scale 2D image collections. To address this limitation, we propose SpatialForge, a scalable data synthesis pipeline that transforms in-the-wild 2D images into spatial reasoning supervision. Our approach decomposes spatial reasoning into perception and relation, and constructs structured supervision signals covering depth, layout, and viewpoint-dependent reasoning, with automatic verification to ensure data quality. Based on this pipeline, we build SpatialForge-10M, a large-scale dataset containing 10 million spatial QA pairs. Extensive experiments across multiple spatial reasoning benchmarks demonstrate that training on SpatialForge-10M significantly improves the spatial reasoning ability of standard VLMs, highlighting the effectiveness of scaling 2D data for 3D-aware spatial reasoning.

Related papers

• SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video [88.79625979053873]
  We propose a novel framework, SpaceEra, for pre-trained vision-language models (VLMs)<n>We extend the original framework into a comprehensive system, termed SpaceEra++, which spans data construction, model design, training optimization, and prompting inference.<n>In addition, to enhance spatial reasoning, we develop SpaceAlign, which enforces pairwise object constraints by jointly exploiting absolute coordinates and relative spatial relations.
  arXiv  Detail & Related papers  (2026-07-02T06:56:29Z)
• SpatialSV: Internalizing Interpretable 3D Spatial Awareness in MLLMs via Task-Oriented Visual Supervision [12.336413790708738]
  spatial intelligence of large language models (MLLMs) is crucial for understanding and interacting with the 3D world.<n>We propose SpatialSV, a framework designed to internalize robust 3D spatial awareness within MLLMs while simultaneously offering inherent interpretability.
  arXiv  Detail & Related papers  (2026-06-18T08:09:32Z)
• Holi-Spatial: Evolving Video Streams into Holistic 3D Spatial Intelligence [78.1406635199656]
  Holi-Spatial is the first fully automated, large-scale, spatially-aware multimodal dataset constructed from raw video inputs without human intervention.<n>Holi-Spatial-4M is the first large-scale, high-quality 3D semantic dataset, containing 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs.
  arXiv  Detail & Related papers  (2026-03-08T14:49:20Z)
• SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards [37.39035418889281]
  We introduce SpatialThinker, a 3D-aware MLLM trained with RL to integrate structured spatial grounding with multi-step reasoning.<n>The model simulates human-like spatial perception by constructing a scene graph of task-relevant objects and spatial relations, and reasoning towards an answer via dense spatial rewards.
  arXiv  Detail & Related papers  (2025-11-10T18:52:47Z)
• RS3DBench: A Comprehensive Benchmark for 3D Spatial Perception in Remote Sensing [71.75704516333394]
  We present a visual Benchmark for 3D understanding of Remotely Sensed images, dubbed RS3DBench.<n>This dataset encompasses 54,951 pairs of remote sensing images and pixel-level aligned depth maps.<n>We introduce a remotely sensed depth estimation model derived from stable diffusion, harnessing its multimodal fusion capabilities.
  arXiv  Detail & Related papers  (2025-09-23T11:20:51Z)
• Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting [64.64738535860351]
  We present a scalable pipeline that converts single-view images into comprehensive, scale- and appearance-realistic 3D representations.<n>Our method bridges the gap between the vast repository of imagery and the increasing demand for spatial scene understanding.<n>By automatically generating authentic, scale-aware 3D data from images, we significantly reduce data collection costs and open new avenues for advancing spatial intelligence.
  arXiv  Detail & Related papers  (2025-07-24T14:53:26Z)
• SURPRISE3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes [105.8644620467576]
  We introduce Stextscurprise3D, a novel dataset designed to evaluate language-guided spatial reasoning segmentation in complex 3D scenes.<n>Stextscurprise3D consists of more than 200k vision language pairs across 900+ detailed indoor scenes from ScanNet++ v2.<n>The dataset contains 89k+ human-annotated spatial queries deliberately crafted without object name.
  arXiv  Detail & Related papers  (2025-07-10T14:01:24Z)
• From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D [32.547597353581594]
  We introduce a novel 2D spatial data generation and annotation pipeline built upon scene data with 3D ground-truth.<n>We construct SPAR-7M, a large-scale dataset generated from thousands of scenes across multiple public datasets.<n>In addition, we introduce SPAR-Bench, a benchmark designed to offer a more comprehensive evaluation of spatial capabilities.
  arXiv  Detail & Related papers  (2025-03-29T04:51:50Z)
• Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes [65.22070581594426]
  "Implicit-Zoo" is a large-scale dataset requiring thousands of GPU training days to facilitate research and development in this field. We showcase two immediate benefits as it enables to: (1) learn token locations for transformer models; (2) directly regress 3D cameras poses of 2D images with respect to NeRF models. This in turn leads to an improved performance in all three task of image classification, semantic segmentation, and 3D pose regression, thereby unlocking new avenues for research.
  arXiv  Detail & Related papers  (2024-06-25T10:20:44Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan.<n>The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.