Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding

• URL: http://arxiv.org/abs/2604.26261v1
• Date: Wed, 29 Apr 2026 03:38:56 GMT
• Title: Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding
• Abstract summary: MCM-VG is a novel framework that achieves robust zero-shot 3DVG.<n>It enforces 2D-3D consistency across three fundamental dimensions.<n> MCM-VG sets a new state-of-the-art for zero-shot 3D visual grounding.
• Score: 26.63197319187971
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot 3D Visual Grounding (3DVG) is a critical capability for open-world embodied AI. However, existing methods are fundamentally bottlenecked by the poor quality of open-vocabulary 3D proposals, suffering from inaccurate categories and imprecise geometries, as well as the spatial redundancy of exhaustive multi-view reasoning. To address these challenges, we propose MCM-VG, a novel framework that achieves robust zero-shot 3DVG by explicitly establishing Multiple Consistent 2D-3D Mappings. Instead of passively relying on noisy 3D segments, MCM-VG enforces 2D-3D consistency across three fundamental dimensions to achieve precise target localization and reliable reasoning. First, a Semantic Alignment module corrects category mismatches via LLM-driven query parsing and coarse-to-fine 2D-3D matching. Second, an Instance Rectification module leverages VLM-guided 2D segmentations to reconstruct missing targets, back-projecting these reliable visual priors to establish accurate 3D geometries. Finally, to eliminate spatial redundancy, a Viewpoint Distillation module clusters 3D camera directions to extract optimal frames. By pairing these optimal RGB frames with Bird's Eye View maps into concise visual prompt sets, we formulate the final target disambiguation as a multiple-choice reasoning task for Vision-Language Models. Extensive evaluations on ScanRefer and Nr3D benchmarks demonstrate that MCM-VG sets a new state-of-the-art for zero-shot 3D visual grounding. Remarkably, it achieves 62.0\% and 53.6\% in Acc@0.25 and Acc@0.5 on ScanRefer, outperforming previous baselines by substantial margins of 6.4\% and 4.0\%.

Related papers

• AgentGrounder: Zero-Shot 3D Visual Pointcloud Grounding using Multimodal Language Models [0.0]
  3D Visual Grounding (3DVG) is an essential capability for embodied AI, requiring agents to localize objects in 3D scenes based on natural language descriptions.<n>We present $bftextAgentGrounder$, a zero-shot 3D visual grounding framework that operates directly on colored point clouds without task-specific 3D training.
  arXiv  Detail & Related papers  (2026-05-25T14:29:04Z)
• Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Grounding [34.1504914582344]
  3D Visual Grounding aims to localize objects in 3D scenes via natural language descriptions.<n>We propose "Think, Act, Build (TAB)", a generative 2D-to-3D reconstruction paradigm operating directly on raw RGB-D streams.<n>To overcome the multi-view coverage deficit caused by strict VLM semantic tracking, we introduce the Semantic-Anchored Geometric Expansion.
  arXiv  Detail & Related papers  (2026-04-01T06:12:16Z)
• Zero-Shot 3D Visual Grounding from Vision-Language Models [10.81711535075112]
  3D Visual Grounding (3DVG) seeks to locate target objects in 3D scenes using natural language descriptions.<n>We present SeeGround, a zero-shot 3DVG framework that leverages 2D Vision-Language Models (VLMs) to bypass the need for 3D-specific training.
  arXiv  Detail & Related papers  (2025-05-28T14:53:53Z)
• DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation [51.43837087865105]
  Vision foundation models (VFMs) trained on large-scale image datasets provide high-quality features that have significantly advanced 2D visual recognition.<n>Their potential in 3D vision remains largely untapped, despite the common availability of 2D images alongside 3D point cloud datasets.<n>We introduce DITR, a simple yet effective approach that extracts 2D foundation model features, projects them to 3D, and finally injects them into a 3D point cloud segmentation model.
  arXiv  Detail & Related papers  (2025-03-24T17:59:11Z)
• SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding [10.81711535075112]
  3D Visual Grounding aims to locate objects in 3D scenes based on textual descriptions, essential for applications like augmented reality and robotics.<n>We introduce SeeGround, a zero-shot 3DVG framework leveraging 2D Vision-Language Models (VLMs) trained on large-scale 2D data.<n>SeeGround represents 3D scenes as a hybrid of query-aligned rendered images and spatially enriched text descriptions, bridging the gap between 3D data and 2D-VLMs input formats.
  arXiv  Detail & Related papers  (2024-12-05T17:58:43Z)
• CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds [55.44204039410225]
  We present a novel two-stage fully sparse convolutional 3D object detection framework, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels. To recover the features of missed voxels due to incorrect voxel-wise segmentation, we build a fully sparse convolutional RoI pooling module.
  arXiv  Detail & Related papers  (2022-10-09T13:38:48Z)
• Multi-initialization Optimization Network for Accurate 3D Human Pose and Shape Estimation [75.44912541912252]
  We propose a three-stage framework named Multi-Initialization Optimization Network (MION) In the first stage, we strategically select different coarse 3D reconstruction candidates which are compatible with the 2D keypoints of input sample. In the second stage, we design a mesh refinement transformer (MRT) to respectively refine each coarse reconstruction result via a self-attention mechanism. Finally, a Consistency Estimation Network (CEN) is proposed to find the best result from mutiple candidates by evaluating if the visual evidence in RGB image matches a given 3D reconstruction.
  arXiv  Detail & Related papers  (2021-12-24T02:43:58Z)
• Multi-Modality Task Cascade for 3D Object Detection [22.131228757850373]
  Many methods train two models in isolation and use simple feature concatenation to represent 3D sensor data. We propose a novel Multi-Modality Task Cascade network (MTC-RCNN) that leverages 3D box proposals to improve 2D segmentation predictions. We show that including a 2D network between two stages of 3D modules significantly improves both 2D and 3D task performance.
  arXiv  Detail & Related papers  (2021-07-08T17:55:01Z)
• Weakly Supervised Volumetric Image Segmentation with Deformed Templates [80.04326168716493]
  We propose an approach that is truly weakly-supervised in the sense that we only need to provide a sparse set of 3D point on the surface of target objects. We will show that it outperforms a more traditional approach to weak-supervision in 3D at a reduced supervision cost.
  arXiv  Detail & Related papers  (2021-06-07T22:09:34Z)
• FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection [78.00922683083776]
  It is non-trivial to make a general adapted 2D detector work in this 3D task. In this technical report, we study this problem with a practice built on fully convolutional single-stage detector. Our solution achieves 1st place out of all the vision-only methods in the nuScenes 3D detection challenge of NeurIPS 2020.
  arXiv  Detail & Related papers  (2021-04-22T09:35:35Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.