SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness

• URL: http://arxiv.org/abs/2604.26341v1
• Date: Wed, 29 Apr 2026 06:46:59 GMT
• Title: SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness
• Abstract summary: We propose a novel framework that internalizes 3D geometric awareness into unified image generation models.<n>Specifically, we first employ a Mixture-of-Transformers (MoT) architecture to augment the MLLM with a parallel spatial transformer.<n>By sharing self-attention with the MLLM, the spatial transformer learns to derive metric-depth maps of target images from rich semantic contexts.<n>These explicit geometric scaffolds are then injected into the diffusion backbone through a specialized depth adapter.
• Score: 62.38746300197673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent unified image generation models have achieved remarkable success by employing MLLMs for semantic understanding and diffusion backbones for image generation. However, these models remain fundamentally limited in spatially-aware tasks due to a lack of intrinsic spatial understanding and the absence of explicit geometric guidance during generation. In this paper, we propose SpatialFusion, a novel framework that internalizes 3D geometric awareness into unified image generation models. Specifically, we first employ a Mixture-of-Transformers (MoT) architecture to augment the MLLM with a parallel spatial transformer to enhance 3D geometric modeling capability. By sharing self-attention with the MLLM, the spatial transformer learns to derive metric-depth maps of target images from rich semantic contexts. These explicit geometric scaffolds are then injected into the diffusion backbone through a specialized depth adapter, providing precise spatial constraints for spatially-coherent image generation. Through a progressive two-stage training strategy, SpatialFusion significantly enhances performance on spatially-aware benchmarks, notably outperforming leading models such as GPT-4o. Additionally, it achieves generalized performance gains across both text-to-image generation and image editing scenarios, all while maintaining negligible inference overhead.

Related papers

• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models [32.477423778941386]
  We present GeoVR, a framework that learns geometric representations using purely 2D video sequences.<n>Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge.<n>Experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-06-04T08:11:12Z)
• Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images [81.94999489820974]
  UniSplat is a feed-forward framework for learning 3D representations from unposed multi-view images.<n>We introduce a dual-masking strategy that strengthens geometry induction in the encoder.<n>Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies.<n>Third, we introduce a pose-conditioned recalibration mechanism that interrelates predicted 3D point and semantic maps into the image plane.
  arXiv  Detail & Related papers  (2026-04-12T10:36:18Z)
• SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning [22.547972947051765]
  Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems.<n>We propose SpatialStack, a hierarchical fusion framework that progressively aligns vision, geometry, and language representations across the model hierarchy.<n>Building upon this framework, we develop VLM-SpatialStack, a model that achieves state-of-the-art performance on multiple 3D spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-28T22:49:40Z)
• GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models [70.61152292499737]
  We argue that this gap does not arise from insufficient geometric priors, but from a misalignment in the training paradigm.<n>Existing approaches typically resort to naive feature concatenation and optimize directly for downstream tasks without geometry-specific supervision.<n>We propose GAP-MLLM, a Geometry-Aligned Pre-training paradigm that explicitly activates structural perception before downstream adaptation.
  arXiv  Detail & Related papers  (2026-03-17T12:43:48Z)
• GACO-CAD: Geometry-Augmented and Conciseness-Optimized CAD Model Generation from Single Image [11.612167656421079]
  Multi-modal large language models (MLLMs) still struggle with accurately inferring 3D geometry from 2D images.<n>We introduce GACO-CAD, a novel two-stage post-training framework.<n>Experiments on the DeepCAD and Fusion360 datasets show that GACO-CAD achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-10-20T04:57:20Z)
• BIFRÖST: 3D-Aware Image compositing with Language Instructions [27.484947109237964]
  Bifr"ost is a novel 3D-aware framework that is built upon diffusion models to perform instruction-based image composition. Bifr"ost addresses issues by training MLLM as a 2.5D location predictor and integrating depth maps as an extra condition during the generation process.
  arXiv  Detail & Related papers  (2024-10-24T18:35:12Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models [67.55362046790512]
  Vision-language models lack the ability to correctly reason over spatial relationships. We develop the REVISION framework which improves spatial fidelity in vision-language models. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware models.
  arXiv  Detail & Related papers  (2024-08-05T04:51:46Z)
• GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation [65.33726478659304]
  We introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-06-21T17:49:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.