Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation

• URL: http://arxiv.org/abs/2601.22228v1
• Date: Thu, 29 Jan 2026 19:01:03 GMT
• Title: Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation
• Abstract summary: Vision-Language Models (VLMs) perform well in 2D perception and semantic reasoning compared to their limited understanding of 3D spatial structure.<n>We introduce VRRPI-Bench, a benchmark derived from unlabeled egocentric videos with verbalized annotations of relative camera motion.<n>We also propose VRRPI-Diag, a diagnostic benchmark that isolates individual motion degrees of freedom.
• Score: 37.0801171856967
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) perform well in 2D perception and semantic reasoning compared to their limited understanding of 3D spatial structure. We investigate this gap using relative camera pose estimation (RCPE), a fundamental vision task that requires inferring relative camera translation and rotation from a pair of images. We introduce VRRPI-Bench, a benchmark derived from unlabeled egocentric videos with verbalized annotations of relative camera motion, reflecting realistic scenarios with simultaneous translation and rotation around a shared object. We further propose VRRPI-Diag, a diagnostic benchmark that isolates individual motion degrees of freedom. Despite the simplicity of RCPE, most VLMs fail to generalize beyond shallow 2D heuristics, particularly for depth changes and roll transformations along the optical axis. Even state-of-the-art models such as GPT-5 ($0.64$) fall short of classic geometric baselines ($0.97$) and human performance ($0.92$). Moreover, VLMs exhibit difficulty in multi-image reasoning, with inconsistent performance (best $59.7\%$) when integrating spatial cues across frames. Our findings reveal limitations in grounding VLMs in 3D and multi-view spatial reasoning.

Related papers

• G$^3$VLA: Geometric inductive bias for Vision-Language-Action Models [3.704517635293094]
  Vision-language-action (VLA) models have made rapid progress in generalist robot manipulation.<n>Their visual tokens remain grounded in 2D image coordinates rather than the calibrated geometry of the robot's cameras.<n>We propose G$3$VLA, a camera-aware geometric module that injects calibrated structure into the visual-token stream of a pretrained VLA.
  arXiv  Detail & Related papers  (2026-06-23T12:02:36Z)
• Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs [72.8641426724502]
  3D localization in Multimodal Large Language Models (MLLMs) is limited by camera intrinsic ambiguity.<n>We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables.<n>Our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale.
  arXiv  Detail & Related papers  (2026-05-19T08:30:21Z)
• Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding [34.1504914582344]
  3D Visual Grounding aims to localize objects in 3D scenes via natural language descriptions.<n>We propose "Think, Act, Build (TAB)", a generative 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams.<n>To overcome the multi-view coverage deficit caused by strict VLM semantic tracking, we introduce the Semantic-Anchored Geometric Expansion.
  arXiv  Detail & Related papers  (2026-04-01T06:12:16Z)
• Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understanding [9.774606705982874]
  This paper significantly enhances Large Language Models (MLLMs) with egomotion modality data.<n>By grounding visual content in physical egomotion trajectories, Motion-MLLM can reason about absolute scale and spatial relationships across the scene.<n>Compared to state-of-the-art (SOTA) methods based on video frames and explicit 3D data, Motion-MLLM exhibits similar or even higher accuracy with significantly less overhead.
  arXiv  Detail & Related papers  (2026-03-18T17:42:49Z)
• Geometry-Guided Camera Motion Understanding in VideoLLMs [8.928448861727274]
  Camera motion is a fundamental geometric signal that shapes visual perception and cinematic style.<n>Current video-capable vision-injection models (VideoLLMs) rarely represent it explicitly and often fail on fine-grained motion primitives.<n>We formulate camera motion as constraint-aware multi-label recognition.<n> Experiments demonstrate improved motion recognition and more camera-aware model responses.
  arXiv  Detail & Related papers  (2026-03-13T16:13:09Z)
• LAST: LeArning to Think in Space and Time for Generalist Vision-Language Models [50.50563228383038]
  We propose LAST, short for LeArn to Think in Space and Time, to improve 3D spatial and long video understanding for general vision-language models.<n>We show that LAST brings substantial gains in various benchmarks, including 3 spatial understanding, 4 video understanding, and 3 image understanding tasks.
  arXiv  Detail & Related papers  (2025-11-24T16:13:26Z)
• Video Spatial Reasoning with Object-Centric 3D Rollout [58.12446467377404]
  We propose Object-Centric 3D Rollout (OCR) to enable robust video spatial reasoning.<n>OCR introduces structured perturbations to the 3D geometry of selected objects during training.<n>OCR compels the model to reason holistically across the entire scene.
  arXiv  Detail & Related papers  (2025-11-17T09:53:41Z)
• VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction [86.82819259860186]
  We introduce VLM-3R, a unified framework for Vision-Language Models (VLMs) that incorporates 3D Reconstructive instruction tuning.<n>VLM-3R processes monocular video frames by employing a geometry encoder to derive implicit 3D tokens that represent spatial understanding.
  arXiv  Detail & Related papers  (2025-05-26T17:56:30Z)
• Are Multimodal Large Language Models Ready for Omnidirectional Spatial Reasoning? [66.88619941063048]
  We ask: Are multimodal large language models (MLLMs) ready for omnidirectional spatial reasoning?<n> OSR-Bench is the first benchmark specifically designed for this setting.<n>It includes over 153,000 diverse question-answer pairs grounded in high-fidelity panoramic indoor scene maps.<n>We evaluate eight state-of-the-art MLLMs, including GPT-4o, Gemini 1.5 Pro, and leading open-source models under zero-shot settings.
  arXiv  Detail & Related papers  (2025-05-17T08:48:40Z)
• Focal-PETR: Embracing Foreground for Efficient Multi-Camera 3D Object Detection [11.13693561702228]
  The dominant multi-camera 3D detection paradigm is based on explicit 3D feature construction. Other methods implicitly introduce geometric positional encoding to build the relationship between image tokens and 3D objects. We propose Focal-PETR with instance-guided supervision and spatial alignment module.
  arXiv  Detail & Related papers  (2022-12-11T13:38:54Z)
• 3D Human Pose Estimation in Multi-View Operating Room Videos Using Differentiable Camera Projections [2.486571221735935]
  We propose to directly optimise for localisation in 3D by training 2D CNNs end-to-end based on a 3D loss. Using videos from the MVOR dataset, we show that this end-to-end approach outperforms optimisation in 2D space.
  arXiv  Detail & Related papers  (2022-10-21T09:00:02Z)
• Monocular 3D Object Detection with Depth from Motion [74.29588921594853]
  We take advantage of camera ego-motion for accurate object depth estimation and detection. Our framework, named Depth from Motion (DfM), then uses the established geometry to lift 2D image features to the 3D space and detects 3D objects thereon. Our framework outperforms state-of-the-art methods by a large margin on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-07-26T15:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.