On the Generalization Capacities of MLLMs for Spatial Intelligence

• URL: http://arxiv.org/abs/2603.06704v1
• Date: Thu, 05 Mar 2026 14:46:11 GMT
• Title: On the Generalization Capacities of MLLMs for Spatial Intelligence
• Abstract summary: We argue that RGB-only approaches are fundamentally flawed in their ability to generalize across cameras.<n>We show this leads MLLMs to overfit to the training camera distribution, rather than learning true and generalizable 3D geometric principles.<n>We propose Camera-Aware MLLM framework for spatial MLLMs.
• Score: 72.21075026598761
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) that directly process RGB inputs for tasks like 3D localization and navigation have shown remarkable potential. However, we argue that these RGB-only approaches are fundamentally flawed in their ability to generalize across cameras. By ignoring camera parameters, they entangle an object's physical properties with the camera's perspective, creating an irresolvable ambiguity. We show this leads MLLMs to overfit to the training camera distribution, rather than learning true and generalizable 3D geometric principles. To address this, we propose Camera-Aware MLLM framework for spatial MLLMs. It learns generalizable spatial reasoning by: (i) injecting camera intrinsics via a dense embedding that conditions each visual token; (ii) introducing a camera-aware data augmentation strategy that synthetically varies camera parameters, forcing the model to disentangle camera properties from scene content; and (iii) distilling geometric priors from a 3D vision foundation model. Extensive experiments demonstrate that camera-aware MLLMs substantially outperform their naive counterparts, particularly in cross-camera generalization tests on spatially-grounded tasks, indicating that camera-awareness is not only beneficial but also a prerequisite for robust and generalizable spatial intelligence in MLLMs.

Related papers

• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models [32.477423778941386]
  We present GeoVR, a framework that learns geometric representations using purely 2D video sequences.<n>Rather than employing superficial feature mixing, GeoVR reshapes the internal representations of the MLLM by distilling geometry knowledge.<n>Experiments on spatial reasoning benchmarks demonstrate that GeoVR achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-06-04T08:11:12Z)
• Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs [72.8641426724502]
  3D localization in Multimodal Large Language Models (MLLMs) is limited by camera intrinsic ambiguity.<n>We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables.<n>Our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale.
  arXiv  Detail & Related papers  (2026-05-19T08:30:21Z)
• SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness [62.38746300197673]
  We propose a novel framework that internalizes 3D geometric awareness into unified image generation models.<n>Specifically, we first employ a Mixture-of-Transformers (MoT) architecture to augment the MLLM with a parallel spatial transformer.<n>By sharing self-attention with the MLLM, the spatial transformer learns to derive metric-depth maps of target images from rich semantic contexts.<n>These explicit geometric scaffolds are then injected into the diffusion backbone through a specialized depth adapter.
  arXiv  Detail & Related papers  (2026-04-29T06:46:59Z)
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding [50.098085774845195]
  We propose a paradigm shift by leveraging the implicit spatial prior within large-scale generation models.<n>We introduce GeneEGA-3D (Video Extracted Generative Awareness), a plug-and-play framework that repurposes a pre-trained video diffusion model as a Latent World Simulator.
  arXiv  Detail & Related papers  (2026-03-19T17:59:58Z)
• SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models [13.88629412035865]
  Large vision-language models (VLMs) show strong multimodal understanding but still struggle with 3D spatial reasoning.<n>We propose SpaceMind, a multimodal large language model explicitly designed for spatial reasoning solely from RGB inputs.
  arXiv  Detail & Related papers  (2025-11-28T11:04:21Z)
• Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots [55.43376513158555]
  Camera Depth Models (CDMs) are a simple plugin on daily-use depth cameras.<n>We develop a neural data engine that generates high-quality paired data from simulation by modeling a depth camera's noise pattern.<n>For the first time, our experiments demonstrate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots.
  arXiv  Detail & Related papers  (2025-09-02T17:29:38Z)
• Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence [13.168559963356952]
  We present Spatial-MLLM, a novel framework for visual-based spatial reasoning from purely 2D observations.<n>Our key insight is to unleash the strong structure prior to the feed-forward visual geometry foundation model.<n>A connector then integrates both features into unified visual tokens for enhanced spatial understanding.
  arXiv  Detail & Related papers  (2025-05-29T17:59:04Z)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation [91.94869042117621]
  Reasoning segmentation aims to segment target objects in complex scenes based on human intent and spatial reasoning.<n>Recent multimodal large language models (MLLMs) have demonstrated impressive 2D image reasoning segmentation.<n>We introduce MLLM-For3D, a framework that transfers knowledge from 2D MLLMs to 3D scene understanding.
  arXiv  Detail & Related papers  (2025-03-23T16:40:20Z)
• Boost 3D Reconstruction using Diffusion-based Monocular Camera Calibration [42.67358714902724]
  DM-Calib is a diffusion-based approach for estimating pinhole camera intrinsic parameters from a single input image.<n>We introduce a new image-based representation, termed Camera Image, which losslessly encodes the numerical camera intrinsics.<n>By fine-tuning a stable diffusion model to generate a Camera Image from a single RGB input, we can extract camera intrinsics via a RANSAC operation.
  arXiv  Detail & Related papers  (2024-11-26T09:04:37Z)
• LLMI3D: MLLM-based 3D Perception from a Single 2D Image [77.13869413871028]
  multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks.<n>In this paper, we propose solutions for weak 3D local spatial object perception, poor text-based geometric numerical output, and inability to handle camera focal variations.<n>We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM.
  arXiv  Detail & Related papers  (2024-08-14T10:00:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.