Multi-Task Domain Adaptation for Language Grounding with 3D Objects

• URL: http://arxiv.org/abs/2407.02846v2
• Date: Fri, 5 Jul 2024 08:10:49 GMT
• Title: Multi-Task Domain Adaptation for Language Grounding with 3D Objects
• Abstract summary: We propose a novel method called Domain Adaptation for Language Grounding (DA4LG) with 3D objects. Specifically, the proposed DA4LG consists of a visual adapter module with multi-task learning to realize vision-language alignment. Experimental results demonstrate that DA4LG competitively performs across visual and non-visual language descriptions.
• Score: 25.975947369817487
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The existing works on object-level language grounding with 3D objects mostly focus on improving performance by utilizing the off-the-shelf pre-trained models to capture features, such as viewpoint selection or geometric priors. However, they have failed to consider exploring the cross-modal representation of language-vision alignment in the cross-domain field. To answer this problem, we propose a novel method called Domain Adaptation for Language Grounding (DA4LG) with 3D objects. Specifically, the proposed DA4LG consists of a visual adapter module with multi-task learning to realize vision-language alignment by comprehensive multimodal feature representation. Experimental results demonstrate that DA4LG competitively performs across visual and non-visual language descriptions, independent of the completeness of observation. DA4LG achieves state-of-the-art performance in the single-view setting and multi-view setting with the accuracy of 83.8% and 86.8% respectively in the language grounding benchmark SNARE. The simulation experiments show the well-practical and generalized performance of DA4LG compared to the existing methods. Our project is available at https://sites.google.com/view/da4lg.

Related papers

• Scaling Diverse Language Generation for 3D Visual Grounding [14.535096337338544]
  ViGiL3D++ is a scalable, scene-agnostic method that generates diverse visual grounding queries.<n>We show that it has greater diversity over existing scaled datasets and improves model performance over several 3DVG benchmarks.
  arXiv  Detail & Related papers  (2026-06-18T21:20:20Z)
• 3EED: Ground Everything Everywhere in 3D [64.28938045702077]
  3EED is a multi-platform, multi-modal 3D grounding benchmark featuring RGB and LiDAR data from vehicle, drone, and quadruped platforms.<n>We provide over 128,000 objects and 22,000 validated referring expressions across diverse outdoor scenes -- 10x larger than existing datasets.
  arXiv  Detail & Related papers  (2025-11-03T17:05:22Z)
• A Simple and Better Baseline for Visual Grounding [41.76403278559263]
  We propose a feature selection-based simple yet effective baseline for visual grounding, called FSVG.<n>Specifically, we encapsulate the linguistic and visual modalities into an overall network architecture without complicated iterative procedures.<n>We introduce a similarity-based feature selection mechanism to only exploit language-related visual features for faster prediction.
  arXiv  Detail & Related papers  (2025-10-12T13:06:59Z)
• IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenes [10.139461308573336]
  IRef-VLA is the largest real-world dataset for the referential grounding task consisting of over 11.5K scanned 3D rooms. We aim to provide a resource for 3D scene understanding that aids the development of robust, interactive navigation systems.
  arXiv  Detail & Related papers  (2025-03-20T16:16:10Z)
• AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring [49.78120051062641]
  3D visual grounding aims to correlate a natural language description with the target object within a 3D scene. Existing approaches commonly encounter a shortage of text3D pairs available for training. We propose AugRefer, a novel approach for advancing 3D visual grounding.
  arXiv  Detail & Related papers  (2025-01-16T09:57:40Z)
• ViGiL3D: A Linguistically Diverse Dataset for 3D Visual Grounding [9.289977174410824]
  3D visual grounding involves localizing entities in a 3D scene referred to by natural language text. We introduce Visual Grounding with Diverse Language in 3D (ViGiL3D), a diagnostic dataset for evaluating visual grounding methods against a diverse set of language patterns.
  arXiv  Detail & Related papers  (2025-01-02T17:20:41Z)
• g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks [62.74304008688472]
  Generalizable 3D-Language Feature Fields (g3D-LF) is a 3D representation model pre-trained on large-scale 3D-language dataset for embodied tasks.
  arXiv  Detail & Related papers  (2024-11-26T01:54:52Z)
• GeoGround: A Unified Large Vision-Language Model. for Remote Sensing Visual Grounding [31.01378033872341]
  GeoGround is a novel framework that unifies support for HBB, OBB, and mask RS visual grounding tasks. To support model training, we present refGeo, a large-scale RS visual instruction-following dataset containing 161k image-text pairs.
  arXiv  Detail & Related papers  (2024-11-16T05:12:11Z)
• High-resolution open-vocabulary object 6D pose estimation [30.835921843505123]
  Horyon is an open-vocabulary VLM-based architecture that addresses relative pose estimation between two scenes of an unseen object. We evaluate our model on a benchmark with a large variety of unseen objects across four datasets.
  arXiv  Detail & Related papers  (2024-06-24T07:53:46Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan. The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• Multi-Granularity Language-Guided Multi-Object Tracking [95.91263758294154]
  We propose a new multi-object tracking framework, named LG-MOT, that explicitly leverages language information at different levels of granularity. At inference, our LG-MOT uses the standard visual features without relying on annotated language descriptions. Our LG-MOT achieves an absolute gain of 2.2% in terms of target object association (IDF1 score) compared to the baseline using only visual features.
  arXiv  Detail & Related papers  (2024-06-07T11:18:40Z)
• Grounded 3D-LLM with Referent Tokens [58.890058568493096]
  We propose Grounded 3D-LLM to consolidate various 3D vision tasks within a unified generative framework. The model uses scene referent tokens as special noun phrases to reference 3D scenes. Per-task instruction-following templates are employed to ensure natural and diversity in translating 3D vision tasks into language formats.
  arXiv  Detail & Related papers  (2024-05-16T18:03:41Z)
• ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models [102.63817106363597]
  We build ELEVATER, the first benchmark to compare and evaluate pre-trained language-augmented visual models. It consists of 20 image classification datasets and 35 object detection datasets, each of which is augmented with external knowledge. We will release our toolkit and evaluation platforms for the research community.
  arXiv  Detail & Related papers  (2022-04-19T10:23:42Z)
• Self-supervised 3D Semantic Representation Learning for Vision-and-Language Navigation [30.429893959096752]
  We develop a novel training framework to encode the voxel-level 3D semantic reconstruction into a 3D semantic representation. We construct an LSTM-based navigation model and train it with the proposed 3D semantic representations and BERT language features on vision-language pairs. Experiments show that the proposed approach achieves success rates of 68% and 66% on the validation unseen and test unseen splits of the R2R dataset.
  arXiv  Detail & Related papers  (2022-01-26T07:43:47Z)
• LanguageRefer: Spatial-Language Model for 3D Visual Grounding [72.7618059299306]
  We develop a spatial-language model for a 3D visual grounding problem. We show that our model performs competitively on visio-linguistic datasets proposed by ReferIt3D.
  arXiv  Detail & Related papers  (2021-07-07T18:55:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.