3DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignment

• URL: http://arxiv.org/abs/2505.01809v1
• Date: Sat, 03 May 2025 12:45:00 GMT
• Title: 3DWG: 3D Weakly Supervised Visual Grounding via Category and Instance-Level Alignment
• Authors: Xiaoqi Li, Jiaming Liu, Nuowei Han, Liang Heng, Yandong Guo, Hao Dong, Yang Liu,
• Abstract summary: The 3D weakly-supervised visual grounding task aims to localize oriented 3D boxes in point clouds based on natural language descriptions.<n>This setting presents two primary challenges: category-level ambiguity and instance-level complexity.<n>Our approach achieves state-of-the-art performance on three widely used benchmarks: Nr3D, Sr3D, and ScanRef.
• Score: 21.094495640408063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The 3D weakly-supervised visual grounding task aims to localize oriented 3D boxes in point clouds based on natural language descriptions without requiring annotations to guide model learning. This setting presents two primary challenges: category-level ambiguity and instance-level complexity. Category-level ambiguity arises from representing objects of fine-grained categories in a highly sparse point cloud format, making category distinction challenging. Instance-level complexity stems from multiple instances of the same category coexisting in a scene, leading to distractions during grounding. To address these challenges, we propose a novel weakly-supervised grounding approach that explicitly differentiates between categories and instances. In the category-level branch, we utilize extensive category knowledge from a pre-trained external detector to align object proposal features with sentence-level category features, thereby enhancing category awareness. In the instance-level branch, we utilize spatial relationship descriptions from language queries to refine object proposal features, ensuring clear differentiation among objects. These designs enable our model to accurately identify target-category objects while distinguishing instances within the same category. Compared to previous methods, our approach achieves state-of-the-art performance on three widely used benchmarks: Nr3D, Sr3D, and ScanRef.

Related papers

• Category-Adaptive Cross-Modal Semantic Refinement and Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  We propose a novel category-adaptive cross-modal semantic refinement and transfer (C$2$SRT) framework to explore the semantic correlation.<n>The proposed framework consists of two complementary modules, i.e., intra-category semantic refinement (ISR) module and inter-category semantic transfer (IST) module.<n>Experiments on OV-MLR benchmarks clearly demonstrate that the proposed C$2$SRT framework outperforms current state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-12-09T04:00:18Z)
• Category-Extensible Out-of-Distribution Detection via Hierarchical Context Descriptions [35.20091752343433]
  This work introduces two hierarchical contexts, namely perceptual context and spurious context, to carefully describe the precise category boundary. The two contexts hierarchically construct the precise description for a certain category, which is first roughly classifying a sample to the predicted category. The precise descriptions for those categories within the vision-language framework present a novel application: CATegory-EXtensible OOD detection (CATEX)
  arXiv  Detail & Related papers  (2024-07-23T12:53:38Z)
• Open-Vocabulary Temporal Action Localization using Multimodal Guidance [67.09635853019005]
  OVTAL enables a model to recognize any desired action category in videos without the need to explicitly curate training data for all categories. This flexibility poses significant challenges, as the model must recognize not only the action categories seen during training but also novel categories specified at inference. We introduce OVFormer, a novel open-vocabulary framework extending ActionFormer with three key contributions.
  arXiv  Detail & Related papers  (2024-06-21T18:00:05Z)
• Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance [49.14140194332482]
  We introduce Open3DIS, a novel solution designed to tackle the problem of Open-Vocabulary Instance within 3D scenes. Objects within 3D environments exhibit diverse shapes, scales, and colors, making precise instance-level identification a challenging task.
  arXiv  Detail & Related papers  (2023-12-17T10:07:03Z)
• Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization [27.583517870047487]
  We propose Deep Object Patch rimis (DOPE), which can be trained from multiple views of object instances without any category or semantic object part labels. To train DOPE, we assume access to sparse depths, foreground masks and known cameras, to obtain pixel-level correspondences between views of an object. We find that DOPE can directly be used for low-shot classification of novel categories using local-part matching, and is competitive with and outperforms supervised and self-supervised learning baselines.
  arXiv  Detail & Related papers  (2022-11-28T04:31:53Z)
• ProposalContrast: Unsupervised Pre-training for LiDAR-based 3D Object Detection [114.54835359657707]
  ProposalContrast is an unsupervised point cloud pre-training framework. It learns robust 3D representations by contrasting region proposals. ProposalContrast is verified on various 3D detectors.
  arXiv  Detail & Related papers  (2022-07-26T04:45:49Z)
• Disentangling What and Where for 3D Object-Centric Representations Through Active Inference [4.088019409160893]
  We propose an active inference agent that can learn novel object categories over time. We show that our agent is able to learn representations for many object categories in an unsupervised way. We validate our system in an end-to-end fashion where the agent is able to search for an object at a given pose from a pixel-based rendering.
  arXiv  Detail & Related papers  (2021-08-26T12:49:07Z)
• I^3Net: Implicit Instance-Invariant Network for Adapting One-Stage Object Detectors [64.93963042395976]
  Implicit Instance-Invariant Network (I3Net) is tailored for adapting one-stage detectors. I3Net implicitly learns instance-invariant features via exploiting the natural characteristics of deep features in different layers. Experiments reveal that I3Net exceeds the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2021-03-25T11:14:36Z)
• Fine-Grained 3D Shape Classification with Hierarchical Part-View Attentions [70.0171362989609]
  We propose a novel fine-grained 3D shape classification method named FG3D-Net to capture the fine-grained local details of 3D shapes from multiple rendered views. Our results under the fine-grained 3D shape dataset show that our method outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-05-26T06:53:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.