Understanding Depth and Height Perception in Large Visual-Language Models

• URL: http://arxiv.org/abs/2408.11748v5
• Date: Fri, 25 Apr 2025 15:56:39 GMT
• Title: Understanding Depth and Height Perception in Large Visual-Language Models
• Abstract summary: We focus on evaluating the geometric understanding of Vision Language Models (VLMs)<n>We find that although they excel in perceiving basic geometric properties like shape and size, they consistently struggle with depth and height perception.<n>This study aims to pave the way for developing VLMs with enhanced geometric understanding.
• Score: 21.209275651704758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Geometric understanding - including depth and height perception - is fundamental to intelligence and crucial for navigating our environment. Despite the impressive capabilities of large Vision Language Models (VLMs), it remains unclear how well they possess the geometric understanding required for practical applications in visual perception. In this work, we focus on evaluating the geometric understanding of these models, specifically targeting their ability to perceive the depth and height of objects in an image. To address this, we introduce GeoMeter, a suite of benchmark datasets - encompassing 2D and 3D scenarios - to rigorously evaluate these aspects. By benchmarking 18 state-of-the-art VLMs, we found that although they excel in perceiving basic geometric properties like shape and size, they consistently struggle with depth and height perception. Our analysis reveal that these challenges stem from shortcomings in their depth and height reasoning capabilities and inherent biases. This study aims to pave the way for developing VLMs with enhanced geometric understanding by emphasizing depth and height perception as critical components necessary for real-world applications.

Related papers

• Make Geometry Matter for Spatial Reasoning [62.61667611352403]
  Vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited.<n>Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs.<n>We propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens.
  arXiv  Detail & Related papers  (2026-03-27T17:45:12Z)
• GeoHeight-Bench: Towards Height-Aware Multimodal Reasoning in Remote Sensing [15.118294966433401]
  We introduce a comprehensive evaluation framework dedicated to height-aware remote sensing understanding.<n>We develop a scalable, VLM-driven data generation pipeline utilizing systematic prompt engineering and metadata extraction.<n>To validate the necessity of height perception, we propose GeoHeightChat, the first height-aware remote sensing LMM baseline.
  arXiv  Detail & Related papers  (2026-03-26T15:38:02Z)
• Doctoral Thesis: Geometric Deep Learning For Camera Pose Prediction, Registration, Depth Estimation, and 3D Reconstruction [1.8782750537161614]
  The dissertation provides solutions to the fundamental challenges in 3D vision.<n>It develops geometric deep learning methods tailored for essential tasks such as camera pose estimation, point cloud registration, depth prediction, and 3D reconstruction.<n>It demonstrates their effectiveness across real-world applications such as digital cultural heritage preservation and immersive VR/AR environments.
  arXiv  Detail & Related papers  (2025-09-02T01:35:44Z)
• MATHGLANCE: Multimodal Large Language Models Do Not Know Where to Look in Mathematical Diagrams [65.02628814094639]
  Diagrams serve as a fundamental form of visual language, representing complex concepts and their inter-relationships through structured symbols, shapes, and spatial arrangements. Current benchmarks conflate perceptual and reasoning tasks, making it difficult to assess whether Multimodal Large Language Models genuinely understand mathematical diagrams beyond superficial pattern recognition. We introduce MATHGLANCE, a benchmark specifically designed to isolate and evaluate mathematical perception in MLLMs. We construct GeoPeP, a perception-oriented dataset of 200K structured geometry image-text annotated with geometric primitives and precise spatial relationships.
  arXiv  Detail & Related papers  (2025-03-26T17:30:41Z)
• Survey on Monocular Metric Depth Estimation [0.9790236766474202]
  Deep learning methods typically estimate relative depth from a single image, but the lack of metric scale often leads to geometric inconsistencies. Monocular Metric Depth Estimation (MMDE) addresses this issue by producing depth maps with absolute scale. This paper presents a structured survey of depth estimation methods, tracing the evolution from traditional geometry-based approaches to modern deep learning models.
  arXiv  Detail & Related papers  (2025-01-21T02:51:10Z)
• Space3D-Bench: Spatial 3D Question Answering Benchmark [49.259397521459114]
  We present Space3D-Bench - a collection of 1000 general spatial questions and answers related to scenes of the Replica dataset. We provide an assessment system that grades natural language responses based on predefined ground-truth answers. Finally, we introduce a baseline called RAG3D-Chat integrating the world understanding of foundation models with rich context retrieval.
  arXiv  Detail & Related papers  (2024-08-29T16:05:22Z)
• GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs [3.2688425993442696]
  The ability to understand and reason about spatial relationships between objects in images is an important component of visual reasoning. We extend the previously released What'sUp dataset and propose a novel comprehensive evaluation for spatial relationship understanding.
  arXiv  Detail & Related papers  (2024-06-19T06:15:26Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan. The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning [17.61621287003562]
  We evaluate vision language models (VLMs) along various axes through the lens of geometry problems. We procedurally create a synthetic dataset of geometry questions with controllable difficulty levels along multiple axes. The empirical results obtained using our benchmark for state-of-the-art VLMs indicate that these models are not as capable in subjects like geometry.
  arXiv  Detail & Related papers  (2023-12-19T15:25:39Z)
• Weakly Supervised Relative Spatial Reasoning for Visual Question Answering [38.05223339919346]
  We evaluate the faithfulness of V&L models to such geometric understanding. We train V&L with weak supervision from off-the-shelf depth estimators. This leads to considerable improvements in accuracy for the "GQA" visual question answering challenge.
  arXiv  Detail & Related papers  (2021-09-04T21:29:06Z)
• Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection [70.71934539556916]
  We learn geometry-guided depth estimation with projective modeling to advance monocular 3D object detection. Specifically, a principled geometry formula with projective modeling of 2D and 3D depth predictions in the monocular 3D object detection network is devised. Our method remarkably improves the detection performance of the state-of-the-art monocular-based method without extra data by 2.80% on the moderate test setting.
  arXiv  Detail & Related papers  (2021-07-29T12:30:39Z)
• DONet: Learning Category-Level 6D Object Pose and Size Estimation from Depth Observation [53.55300278592281]
  We propose a method of Category-level 6D Object Pose and Size Estimation (COPSE) from a single depth image. Our framework makes inferences based on the rich geometric information of the object in the depth channel alone. Our framework competes with state-of-the-art approaches that require labeled real-world images.
  arXiv  Detail & Related papers  (2021-06-27T10:41:50Z)
• Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction [87.08227378010874]
  We show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces a simple geometric constraint, we significantly improve the accuracy and robustness of monocular depth estimation. We show state-of-the-art results of learning metric depth on NYU Depth-V2 and KITTI.
  arXiv  Detail & Related papers  (2021-03-07T00:08:21Z)
• Predicting Relative Depth between Objects from Semantic Features [2.127049691404299]
  The 3D depth of objects depicted in 2D images is one such feature. The state of the art in this area are complex Neural Network models trained on stereo image data to predict depth per pixel. An overall increase of 14% in relative depth accuracy over relative depth computed from the monodepth model derived results is achieved.
  arXiv  Detail & Related papers  (2021-01-12T17:28:23Z)
• Learning to Reconstruct and Segment 3D Objects [4.709764624933227]
  We aim to understand scenes and the objects within them by learning general and robust representations using deep neural networks. This thesis makes three core contributions from object-level 3D shape estimation from single or multiple views to scene-level semantic understanding.
  arXiv  Detail & Related papers  (2020-10-19T15:09:04Z)
• Occlusion-Aware Depth Estimation with Adaptive Normal Constraints [85.44842683936471]
  We present a new learning-based method for multi-frame depth estimation from a color video. Our method outperforms the state-of-the-art in terms of depth estimation accuracy.
  arXiv  Detail & Related papers  (2020-04-02T07:10:45Z)
• Learning Depth With Very Sparse Supervision [57.911425589947314]
  This paper explores the idea that perception gets coupled to 3D properties of the world via interaction with the environment. We train a specialized global-local network architecture with what would be available to a robot interacting with the environment. Experiments on several datasets show that, when ground truth is available even for just one of the image pixels, the proposed network can learn monocular dense depth estimation up to 22.5% more accurately than state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-02T10:44:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.