Multimodal Language Models Cannot Spot Spatial Inconsistencies

• URL: http://arxiv.org/abs/2604.00799v2
• Date: Thu, 02 Apr 2026 21:20:30 GMT
• Title: Multimodal Language Models Cannot Spot Spatial Inconsistencies
• Abstract summary: multimodal large language models (MLLMs) often struggle to reason about 3D geometry across multiple views.<n>We introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency.<n>We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes.
• Score: 10.81222944820803
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial consistency is a fundamental property of the visual world and a key requirement for models that aim to understand physical reality. Despite recent advances, multimodal large language models (MLLMs) often struggle to reason about 3D geometry across multiple views. Rather than asking models to describe scene attributes, we introduce a more challenging task: given two views of the same scene, identify the object that violates 3D motion consistency. We propose a simple and scalable method for generating realistic, spatially inconsistent image pairs from multi-view scenes, enabling systematic evaluation of this capability. Our results show that state-of-the-art MLLMs significantly underperform human observers and exhibit substantial variability across different scene attributes, revealing a fragile and incomplete understanding of 3D structure. We hope our findings underscore the need for approaches that develop a more deeply grounded understanding of the physical world.

Related papers

• WorldSculpt: Generating Compositional Worlds from Grounded Videos [50.427204240556854]
  We study the problem of generating a compositional 3D representation of a cluttered scene containing hundreds of objects.<n>This task is challenging in densely cluttered scenes, where objects heavily occlude one another and each view reveals only a fraction of their geometry.<n>We show that complex scenes with hundreds of objects can instead be generated compositionally by adapting a strong single-object 3D generative prior to multi-view observations.
  arXiv  Detail & Related papers  (2026-09-04T17:59:35Z)
• Rendering Multi-Human and Multi-Object with 3D Gaussian Splatting [47.35917581534084]
  Reconstructing dynamic scenes with multiple interacting humans and objects from sparse-view inputs is critical for creating high-fidelity digital twins for robotics and VR/AR.<n>This problem, which we term Multi-Human Multi-Object rendering, presents two significant obstacles.<n>We propose MM-GS, a novel hierarchical framework built upon 3D Splatting.
  arXiv  Detail & Related papers  (2026-04-03T12:20:36Z)
• Unified Semantic Transformer for 3D Scene Understanding [55.415468022487005]
  We introduce UNITE, a novel feed-forward neural network that unifies a diverse set of 3D semantic tasks within a single model.<n>Our model operates on unseen scenes in a fully end-to-end manner and only takes a few seconds to infer the full 3D semantic geometry.<n>We demonstrate that UNITE achieves state-of-the-art performance on several different semantic tasks and even outperforms task-specific models.
  arXiv  Detail & Related papers  (2025-12-16T12:49:35Z)
• Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling [68.14113731953971]
  This paper introduces MILO, an Implicit spatIaL wOrld modeling paradigm that simulates human-like imagination.<n>We show that our approach significantly enhances spatial reasoning capabilities across multiple baselines and benchmarks.
  arXiv  Detail & Related papers  (2025-12-01T16:01:41Z)
• Evo-0: Vision-Language-Action Model with Implicit Spatial Understanding [11.222744122842023]
  We introduce a plug-and-play module that implicitly incorporates 3D geometry features into Vision-Language-Action (VLA) models.<n>Our method significantly improves the performance of state-of-the-art VLA models across diverse scenarios.
  arXiv  Detail & Related papers  (2025-07-01T04:05:47Z)
• Video Perception Models for 3D Scene Synthesis [109.5543506037003]
  VIPScene is a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models.<n>VIPScene seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene.
  arXiv  Detail & Related papers  (2025-06-25T16:40:17Z)
• Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations [61.235500325327585]
  Existing AI benchmarks primarily assess verbal reasoning, neglecting the complexities of non-verbal, multi-step visual simulation.<n>We introduce STARE, a benchmark designed to rigorously evaluate multimodal large language models on tasks better solved through visual simulation.<n>Our evaluations show that models excel at reasoning over simpler 2D transformations, but perform close to random chance on more complex tasks.
  arXiv  Detail & Related papers  (2025-06-05T05:09:46Z)
• MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation [91.94869042117621]
  Reasoning segmentation aims to segment target objects in complex scenes based on human intent and spatial reasoning.<n>Recent multimodal large language models (MLLMs) have demonstrated impressive 2D image reasoning segmentation.<n>We introduce MLLM-For3D, a framework that transfers knowledge from 2D MLLMs to 3D scene understanding.
  arXiv  Detail & Related papers  (2025-03-23T16:40:20Z)
• Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Model [51.83436609094658]
  We introduce Coarse Correspondences, a simple lightweight method that enhances MLLMs' spatial-temporal reasoning with 2D images as input. Our method uses a lightweight tracking model to identify primary object correspondences between frames in a video or across different image viewpoints. We demonstrate that this simple training-free approach brings substantial gains to GPT4-V/O consistently on four benchmarks.
  arXiv  Detail & Related papers  (2024-08-01T17:57:12Z)
• Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views [9.556376932449187]
  Multi-View and Multi-Object Network (MulMON) is a method for learning accurate, object-centric representations of multi-object scenes by leveraging multiple views. We show that MulMON better-resolves spatial ambiguities than single-view methods.
  arXiv  Detail & Related papers  (2021-11-13T13:54:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.