GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding

• URL: http://arxiv.org/abs/2608.00518v1
• Date: Sat, 01 Aug 2026 08:28:41 GMT
• Title: GuideGround: VLM-guided Semantic Understanding and Viewpoint-aware Reasoning for 3D Visual Grounding
• Abstract summary: 3D visual grounding aims to localize the target object in a 3D scene from a natural language query.<n>We propose GuideGround, a VLM-guided framework that complements rather than replaces task-specific grounding models.
• Score: 11.915446160398625
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D visual grounding aims to localize the target object in a 3D scene from a natural language query, requiring both fine-grained semantic understanding and viewpoint-dependent spatial reasoning. Existing methods typically formulate semantic understanding as an auxiliary closed-set object classification task and rely on multi-view feature aggregation for viewpoint reasoning, limiting semantic generalization and weakening viewpoint-specific evidence. We observe that vision-language models naturally provide complementary capabilities through open-vocabulary semantic understanding and global scene perception. Based on this insight, we propose GuideGround, a VLM-guided framework that complements rather than replaces task-specific grounding models by leveraging VLMs for semantic enhancement and viewpoint-specific hypothesis verification. Specifically, we replace auxiliary closed-set object classification with VLM-generated object semantic descriptions to enhance semantic understanding. Meanwhile, instead of directly aggregating multi-view representations, we preserve viewpoint-specific grounding hypotheses through per-view grounding and explicitly verify them using VLMs across candidate viewpoints. Extensive experiments on the ReferIt3D benchmark demonstrate that GuideGround consistently outperforms previous state-of-the-art methods. Comprehensive ablation studies further confirm the effectiveness of both the proposed semantic understanding and viewpoint reasoning strategies.

Related papers

• AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Models [15.021275200375504]
  This study investigates the challenge of ambiguity faced by Vision-Language Models (VLMs) in understanding spatial semantics.<n>Natural language descriptions of spatial relations frequently omit explicit reference frames, leading to semantic ambiguity and potentially serious errors for embodied AI robots.<n>We construct a new dataset, AlloEgo-View, comprising (image, query, view-specific answer) triplets that capture key object relations from both allocentric and egocentric perspectives.<n>We develop AlloEgo-VLM, a framework to disambiguate allocentric and egocentric reference frames, even under ambiguous queries, and to be easily integrated
  arXiv  Detail & Related papers  (2026-08-16T08:06:20Z)
• Beyond Isolated Objects: Relationship-aware Open Vocabulary Scene Understanding via 3D Scene Graph Analysis [59.527737790821305]
  We propose a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding.<n>Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships.<n>Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability.
  arXiv  Detail & Related papers  (2026-07-06T17:29:49Z)
• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models [39.77362541405281]
  EPIC-Bench is a grounding benchmark for large vision-language models (VLMs)<n>It systematically evaluates the visual perceptual capabilities of VLMs in real-world embodied environments.
  arXiv  Detail & Related papers  (2026-05-16T16:38:51Z)
• VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection [12.835071167163607]
  VirPro is an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks.<n>We generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB)
  arXiv  Detail & Related papers  (2026-03-18T08:23:55Z)
• Thinking with Blueprints: Assisting Vision-Language Models in Spatial Reasoning via Structured Object Representation [52.605647992080485]
  spatial reasoning advances vision-language models from visual perception toward semantic understanding.<n>We integrate the cognitive concept of an object-centric blueprint into spatial reasoning.<n>Our method consistently outperforms existing vision-language models.
  arXiv  Detail & Related papers  (2026-01-05T10:38:26Z)
• Point What You Mean: Visually Grounded Instruction Policy [42.52502990975079]
  Point-VLA is a plug-and-play policy that augments language instructions with explicit visual cues to resolve referential ambiguity.<n>We evaluate Point-VLA on diverse real-world referring tasks and observe consistently stronger performance than text-only instruction VLAs.
  arXiv  Detail & Related papers  (2025-12-22T00:44:19Z)
• A Multimodal Depth-Aware Method For Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding requires identifying a target object in a visual scene based on both language instructions and pointing cues.<n>We propose a novel ERU framework that jointly leverages data augmentation, depth-map modality, and a depth-aware decision module.
  arXiv  Detail & Related papers  (2025-10-09T14:32:21Z)
• ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoning [68.4209681278336]
  Open-vocabulary 3D visual grounding and reasoning aim to localize objects in a scene based on implicit language descriptions.<n>Current methods struggle because they rely heavily on fine-tuning with 3D annotations and mask proposals.<n>We propose ReasonGrounder, an LVLM-guided framework that uses hierarchical 3D feature Gaussian fields for adaptive grouping.
  arXiv  Detail & Related papers  (2025-03-30T03:40:35Z)
• Oh-A-DINO: Understanding and Enhancing Attribute-Level Information in Self-Supervised Object-Centric Representations [9.949149600332836]
  Self-supervised vision models and slot-based representations excel at identifying edge-derived geometry but fail to preserve non-geometric surface-level cues.<n>We show that learning an auxiliary latent space over segmented patches, where VAE regularisation enforces compact, disentangled object-centric representations, recovers these missing attributes.
  arXiv  Detail & Related papers  (2025-03-12T21:57:41Z)
• VLScene: Vision-Language Guidance Distillation for Camera-Based 3D Semantic Scene Completion [35.34118012715217]
  Camera-based 3D semantic scene completion (SSC) provides dense geometric and semantic perception for autonomous driving.<n>Existing methods often lack explicit semantic modeling between objects, limiting their perception of 3D semantic context.<n>We propose a novel method VLScene: Vision-Language Guidance Distillation for Camera-based 3D Semantic Scene Completion.
  arXiv  Detail & Related papers  (2025-03-08T13:40:52Z)
• LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models [57.92316645992816]
  Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space.<n>We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs)<n>We demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.
  arXiv  Detail & Related papers  (2024-12-03T06:15:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.