LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Grounding

• URL: http://arxiv.org/abs/2505.04058v3
• Date: Fri, 15 Aug 2025 03:24:08 GMT
• Title: LSVG: Language-Guided Scene Graphs with 2D-Assisted Multi-Modal Encoding for 3D Visual Grounding
• Authors: Feng Xiao, Hongbin Xu, Guocan Zhao, Wenxiong Kang,
• Abstract summary: 3D visual grounding aims to localize the unique target described by natural languages in 3D scenes.<n>We propose a novel 3D visual grounding framework that constructs language-guided scene graphs with referred object discrimination.
• Score: 15.944945244005952
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: 3D visual grounding aims to localize the unique target described by natural languages in 3D scenes. The significant gap between 3D and language modalities makes it a notable challenge to distinguish multiple similar objects through the described spatial relationships. Current methods attempt to achieve cross-modal understanding in complex scenes via a target-centered learning mechanism, ignoring the modeling of referred objects. We propose a novel 3D visual grounding framework that constructs language-guided scene graphs with referred object discrimination to improve relational perception. The framework incorporates a dual-branch visual encoder that leverages pre-trained 2D semantics to enhance and supervise the multi-modal 3D encoding. Furthermore, we employ graph attention to promote relationship-oriented information fusion in cross-modal interaction. The learned object representations and scene graph structure enable effective alignment between 3D visual content and textual descriptions. Experimental results on popular benchmarks demonstrate our superior performance compared to state-of-the-art methods, especially in handling the challenges of multiple similar distractors.

Related papers

• Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions [28.185661905201222]
  Descrip3D is a novel framework that explicitly encodes the relationships between objects using natural language.<n>It allows for unified reasoning across various tasks such as grounding, captioning, and question answering.
  arXiv  Detail & Related papers  (2025-07-19T09:19:16Z)
• NVSMask3D: Hard Visual Prompting with Camera Pose Interpolation for 3D Open Vocabulary Instance Segmentation [14.046423852723615]
  We introduce a novel 3D Gaussian Splatting based hard visual prompting approach to generate diverse viewpoints around target objects.<n>Our method simulates realistic 3D perspectives, effectively augmenting existing hard visual prompts.<n>This training-free strategy integrates seamlessly with prior hard visual prompts, enriching object-descriptive features.
  arXiv  Detail & Related papers  (2025-04-20T14:39:27Z)
• AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring [49.78120051062641]
  3D visual grounding aims to correlate a natural language description with the target object within a 3D scene.<n>Existing approaches commonly encounter a shortage of text3D pairs available for training.<n>We propose AugRefer, a novel approach for advancing 3D visual grounding.
  arXiv  Detail & Related papers  (2025-01-16T09:57:40Z)
• 3D Scene Graph Guided Vision-Language Pre-training [11.131667398927394]
  3D vision-language (VL) reasoning has gained significant attention due to its potential to bridge the 3D physical world with natural language descriptions.<n>Existing approaches typically follow task-specific, highly specialized paradigms.<n>This paper proposes a 3D scene graph-guided vision-language pre-training framework.
  arXiv  Detail & Related papers  (2024-11-27T16:10:44Z)
• Grounding 3D Scene Affordance From Egocentric Interactions [52.5827242925951]
  Grounding 3D scene affordance aims to locate interactive regions in 3D environments. We introduce a novel task: grounding 3D scene affordance from egocentric interactions.
  arXiv  Detail & Related papers  (2024-09-29T10:46:19Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• SeCG: Semantic-Enhanced 3D Visual Grounding via Cross-modal Graph Attention [19.23636231942245]
  We propose a semantic-enhanced relational learning model based on a graph network with our designed memory graph attention layer. Our method replaces original language-independent encoding with cross-modal encoding in visual analysis. Experimental results on ReferIt3D and ScanRefer benchmarks show that the proposed method outperforms the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-13T02:11:04Z)
• Weakly-Supervised 3D Visual Grounding based on Visual Language Alignment [24.63428589906294]
  We propose a weakly supervised approach for 3D visual grounding based on Visual Linguistic Alignment.<n>Our 3D-VLA exploits the superior ability of current large-scale vision-language models on aligning the semantics between texts and 2D images.<n>During the inference stage, the learned text-3D correspondence will help us ground the text queries to the 3D target objects even without 2D images.
  arXiv  Detail & Related papers  (2023-12-15T09:08:14Z)
• TeMO: Towards Text-Driven 3D Stylization for Multi-Object Meshes [67.5351491691866]
  We present a novel framework, dubbed TeMO, to parse multi-object 3D scenes and edit their styles. Our method can synthesize high-quality stylized content and outperform the existing methods over a wide range of multi-object 3D meshes.
  arXiv  Detail & Related papers  (2023-12-07T12:10:05Z)
• Four Ways to Improve Verbo-visual Fusion for Dense 3D Visual Grounding [56.00186960144545]
  3D visual grounding is the task of localizing the object in a 3D scene which is referred by a description in natural language. We propose a dense 3D grounding network, featuring four novel stand-alone modules that aim to improve grounding performance.
  arXiv  Detail & Related papers  (2023-09-08T19:27:01Z)
• Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding [57.47315482494805]
  Open-world instance-level scene understanding aims to locate and recognize unseen object categories that are not present in the annotated dataset. This task is challenging because the model needs to both localize novel 3D objects and infer their semantic categories. We propose to harness pre-trained vision-language (VL) foundation models that encode extensive knowledge from image-text pairs to generate captions for 3D scenes.
  arXiv  Detail & Related papers  (2023-08-01T07:50:14Z)
• 3DRP-Net: 3D Relative Position-aware Network for 3D Visual Grounding [58.924180772480504]
  3D visual grounding aims to localize the target object in a 3D point cloud by a free-form language description. We propose a relation-aware one-stage framework, named 3D Relative Position-aware Network (3-Net)
  arXiv  Detail & Related papers  (2023-07-25T09:33:25Z)
• Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes [68.61199623705096]
  Training models to apply common-sense linguistic knowledge and visual concepts from 2D images to 3D scene understanding is a promising direction that researchers have only recently started to explore. We propose a novel 3D pre-training Vision-Language method, namely Multi-CLIP, that enables a model to learn language-grounded and transferable 3D scene point cloud representations.
  arXiv  Detail & Related papers  (2023-06-04T11:08:53Z)
• Grounding 3D Object Affordance from 2D Interactions in Images [128.6316708679246]
  Grounding 3D object affordance seeks to locate objects' ''action possibilities'' regions in the 3D space. Humans possess the ability to perceive object affordances in the physical world through demonstration images or videos. We devise an Interaction-driven 3D Affordance Grounding Network (IAG), which aligns the region feature of objects from different sources.
  arXiv  Detail & Related papers  (2023-03-18T15:37:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.