Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation

• URL: http://arxiv.org/abs/2606.03100v2
• Date: Thu, 04 Jun 2026 05:13:15 GMT
• Title: Zero-Shot 3D Question Answering via Hierarchical View-to-Token Transportation
• Abstract summary: We propose a hierarchical approach for input context collection at both the view and token levels.<n>Specifically, we combine pixel features with camera parameters and assess view importance based on both semantic content and geometric position.<n>We evaluate our framework on three widely used benchmarks, demonstrating significant improvements over existing tuning-free methods and performance comparable to training-based approaches.
• Score: 9.296275675671636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, zero-shot 3D scene understanding via 2D Vision-Language Models (VLMs) has gained increasing research interest due to their promising spatial reasoning capabilities. Typically, multiple 2D views are sampled from a 3D point cloud and fed into pre-trained VLMs to answer a given question. This paradigm highlights the critical role of input context quality and raises the challenge of retaining as many task-relevant 3D details as possible under a limited input budget. We propose \texttt{KeyVT}, a hierarchical approach for input context collection at both the view and token levels. Specifically, we combine pixel features with camera parameters and assess view importance based on both semantic content and geometric position, resulting in spatially consistent and task-relevant views. Furthermore, we address redundancy among patches across selected views by identifying representative tokens under the optimal transport (OT) framework, where view tokens and key tokens are formulated as two discrete distributions in the embedding space. These key tokens are expected to cover all view features by minimizing the OT distance. We evaluate our framework on three widely used benchmarks, demonstrating significant improvements over existing tuning-free methods and performance comparable to training-based approaches.

Related papers

• Agentic Collaborative Cognition for Zero-Shot 3D Understanding [57.854375071638856]
  We propose a collaborative multi-agent framework that assigns a Planning Agent to handle high-level viewpoint planning.<n>Perception Agent to explicitly summarize the 3D scene into a structured holistic cognitive map.<n>Our method achieves state-of-the-art performance on 6 benchmarks, with improvements of 11.1% Acc@0.5 on ScanRefer, 14.6 BLEU-1 on 3D-assisted dialog, and 2.1 EM on SQA3D.
  arXiv  Detail & Related papers  (2026-06-23T14:44:04Z)
• CVSBench: A Comprehensive Benchmark for Cross-view Spatial Reasoning and Dreaming [13.534076118011603]
  We introduce CVSBench, a large-scale benchmark for evaluating cross-view spatial reasoning through satellite-street pairs.<n>This benchmark supports multiple tasks, including cross-view VQA, cross-view grounding, and viewpoint identification.<n>Our findings demonstrate that language-only reasoning yields marginal improvements, while incorporating visual spatial imagination via a 3D scene imagination pipeline substantially improves cross-view reasoning.
  arXiv  Detail & Related papers  (2026-06-21T12:35:43Z)
• Cross-Attentive Multiview Fusion of Vision-Language Embeddings [28.984855441235776]
  We introduce a novel multiview transformer architecture that cross-attends across vision-language descriptors from multiple viewpoints.<n>We leverage multiview consistency as a self-supervision signal for this fusion.<n>Our Cross-Attentive Multiview Fusion consistently outperforms naive averaging or single-view descriptor selection.
  arXiv  Detail & Related papers  (2026-04-14T10:25:32Z)
• HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model [14.277165215664425]
  Large vision-language models (VLMs) have shown significant promise for 3D scene understanding.<n>Existing VLM-based approaches typically align 3D scene features with the VLM's embedding space.<n>We propose a novel hierarchical multimodal representation for 3D scene reasoning.
  arXiv  Detail & Related papers  (2025-11-28T08:06:20Z)
• Abstract 3D Perception for Spatial Intelligence in Vision-Language Models [100.13033631690114]
  Vision-language models (VLMs) struggle with 3D-related tasks such as spatial cognition and physical understanding.<n>We introduce SandboxVLM, a framework that leverages abstract bounding boxes to encode geometric structure and physical kinematics for VLM.<n>Our approach consistently improves spatial intelligence, achieving an 8.3% gain on SAT Real compared with baseline methods.
  arXiv  Detail & Related papers  (2025-11-14T04:16:09Z)
• Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding [24.964149224068027]
  We propose Fast3D, a plug-and-play visual token pruning framework for 3D MLLMs.<n>Global Attention Prediction (GAP) learns to predict the global attention distributions of the target model, enabling efficient token importance estimation.<n>SAP, introduces dynamic token budgets through attention-based complexity assessment, automatically adjusting layer-wise pruning ratios.
  arXiv  Detail & Related papers  (2025-07-12T16:29:02Z)
• Pts3D-LLM: Studying the Impact of Token Structure for 3D Scene Understanding With Large Language Models [9.658828841170472]
  This work presents a rigorous study of 3D token structures, systematically comparing video-based and point-based representations.<n>We propose a novel approach that enriches visual tokens by incorporating 3D point cloud features from a Sonata pretrained Point Transformer V3 encoder.
  arXiv  Detail & Related papers  (2025-06-06T02:35:26Z)
• ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models [68.46716645478661]
  Vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and reasoning about visual content.<n>Current VLMs excel primarily at egocentric spatial reasoning (from the camera's perspective) but fail to generalize to allocentric viewpoints.<n>We introduce ViewSpatial-Bench, the first comprehensive benchmark designed specifically for multi-viewpoint spatial localization recognition evaluation.
  arXiv  Detail & Related papers  (2025-05-27T17:59:26Z)
• AR-1-to-3: Single Image to Consistent 3D Object Generation via Next-View Prediction [69.65671384868344]
  We propose AR-1-to-3, a novel next-view prediction paradigm based on diffusion models.<n>We show that our method significantly improves the consistency between the generated views and the input views, producing high-fidelity 3D assets.
  arXiv  Detail & Related papers  (2025-03-17T08:39:10Z)
• UniPLV: Towards Label-Efficient Open-World 3D Scene Understanding by Regional Visual Language Supervision [10.587237925455211]
  We present UniPLV, a robust framework that unifies point clouds, images, and text within a single learning paradigm for comprehensive 3D scene understanding.<n>We show that UniPLV significantly surpasses state-of-the-art methods, with average improvements of 15.6% and 14.8% in semantic segmentation for Base-Annotated and.<n>Free tasks.
  arXiv  Detail & Related papers  (2024-12-24T03:40:05Z)
• VAPO: Visibility-Aware Keypoint Localization for Efficient 6DoF Object Pose Estimation [52.81869878956534]
  Localizing 3D keypoints in a 2D image is an effective way to establish 3D-2D correspondences for instance-level 6DoF object pose estimation.<n>In this paper, we address this issue by localizing the important keypoints in terms of visibility.<n>We construct VAPO (Visibility-Aware POse estimator) by integrating the visibility-aware importance with a state-of-the-art pose estimation algorithm.
  arXiv  Detail & Related papers  (2024-03-21T16:59:45Z)
• CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection [57.44434974289945]
  We propose Contextualized Multi-Stage Refinement for 3D Object Detection (CMR3D) framework. Our framework takes a 3D scene as input and strives to explicitly integrate useful contextual information of the scene. In addition to 3D object detection, we investigate the effectiveness of our framework for the problem of 3D object counting.
  arXiv  Detail & Related papers  (2022-09-13T05:26:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.