Data-Efficient 3D Visual Grounding via Order-Aware Referring

• URL: http://arxiv.org/abs/2403.16539v5
• Date: Sun, 23 Feb 2025 04:04:38 GMT
• Title: Data-Efficient 3D Visual Grounding via Order-Aware Referring
• Authors: Tung-Yu Wu, Sheng-Yu Huang, Yu-Chiang Frank Wang,
• Abstract summary: Vigor is a novel Data-Efficient 3D Visual Grounding framework via Order-aware Referring.<n>We present an order-aware warm-up training strategy, which augments referential orders for pre-training the visual grounding framework.
• Score: 31.96736077210907
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D visual grounding aims to identify the target object within a 3D point cloud scene referred to by a natural language description. Previous works usually require significant data relating to point color and their descriptions to exploit the corresponding complicated verbo-visual relations. In our work, we introduce Vigor, a novel Data-Efficient 3D Visual Grounding framework via Order-aware Referring. Vigor leverages LLM to produce a desirable referential order from the input description for 3D visual grounding. With the proposed stacked object-referring blocks, the predicted anchor objects in the above order allow one to locate the target object progressively without supervision on the identities of anchor objects or exact relations between anchor/target objects. In addition, we present an order-aware warm-up training strategy, which augments referential orders for pre-training the visual grounding framework. This allows us to better capture the complex verbo-visual relations and benefit the desirable data-efficient learning scheme. Experimental results on the NR3D and ScanRefer datasets demonstrate our superiority in low-resource scenarios. In particular, Vigor surpasses current state-of-the-art frameworks by 9.3% and 7.6% grounding accuracy under 1% data and 10% data settings on the NR3D dataset, respectively. Our code is publicly available at https://github.com/tony10101105/Vigor.

Related papers

• Open-Vocabulary Octree-Graph for 3D Scene Understanding [54.11828083068082]
  Octree-Graph is a novel scene representation for open-vocabulary 3D scene understanding. An adaptive-octree structure is developed that stores semantics and depicts the occupancy of an object adjustably according to its shape.
  arXiv  Detail & Related papers  (2024-11-25T10:14:10Z)
• VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding [57.04804711488706]
  3D visual grounding is crucial for robots, requiring integration of natural language and 3D scene understanding. We present VLM-Grounder, a novel framework using vision-language models (VLMs) for zero-shot 3D visual grounding based solely on 2D images.
  arXiv  Detail & Related papers  (2024-10-17T17:59:55Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan. The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• 3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding [58.924180772480504]
  3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description. We propose a relation-aware one-stage framework, named 3D Relative Position-aware Network (3-Net)
  arXiv  Detail & Related papers  (2023-07-25T09:33:25Z)
• Point2Seq: Detecting 3D Objects as Sequences [58.63662049729309]
  We present a simple and effective framework, named Point2Seq, for 3D object detection from point clouds. We view each 3D object as a sequence of words and reformulate the 3D object detection task as decoding words from 3D scenes in an auto-regressive manner.
  arXiv  Detail & Related papers  (2022-03-25T00:20:31Z)
• Free-form Description Guided 3D Visual Graph Network for Object Grounding in Point Cloud [39.055928838826226]
  3D object grounding aims to locate the most relevant target object in a raw point cloud scene based on a free-form language description. We propose a language scene graph module to capture the rich structure and long-distance phrase correlations. Secondly, we introduce a multi-level 3D proposal relation graph module to extract the object-object and object-scene co-occurrence relationships.
  arXiv  Detail & Related papers  (2021-03-30T14:22:36Z)
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding [107.02479689909164]
  In this work, we aim at facilitating research on 3D representation learning. We measure the effect of unsupervised pre-training on a large source set of 3D scenes.
  arXiv  Detail & Related papers  (2020-07-21T17:59:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.