Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

• URL: http://arxiv.org/abs/2608.02980v1
• Date: Tue, 04 Aug 2026 00:32:09 GMT
• Title: Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
• Abstract summary: 3D geometry provides a natural compression mechanism for visual streams.<n>Qwen-3D is a geometry-aware LMM that compresses visual information within the Qwen backbone using multi-view geometric cues.<n>Qwen-3D surpasses existing 3D LMMs and outperforms several large proprietary 2D models.
• Score: 24.745098191502233
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Multimodal Models (LMMs) have achieved remarkable success on images and short videos, yet scaling them to long videos remains challenging due to frame-centric tokenization and limited context windows. 3D geometry provides a natural compression mechanism for visual streams: depth and camera pose enable observations from multiple views and time steps to be fused into a persistent, world-aligned representation. While recent 3D LMMs leverage geometry-aware representations to improve spatial reasoning, they continue to lag behind specialist 3D perception systems on grounding and segmentation tasks. We argue that a key limitation is geometry-aware decoding: existing methods communicate 3D predictions through language tokens, proposal selection, or lightweight grounding queries, creating a bottleneck between language reasoning and dense geometric prediction. Building on these insights, we introduce Qwen-3D, a geometry-aware LMM that compresses visual information within the Qwen backbone using multi-view geometric cues, enabling efficient long-horizon visual reasoning over static scenes. Qwen-3D augments visual tokens with 3D Rotary Positional Embeddings, allowing attention to operate directly in 3D scene space rather than across independent image frames and thereby facilitating scalable cross-view and temporal reasoning. To bridge language and geometry, Qwen-3D incorporates a query-based segmentation decoder that grounds language directly in the underlying 3D scene representation, unifying referential grounding, instance segmentation, and visual question answering across both images and videos. Across a diverse set of benchmarks, Qwen-3D surpasses existing 3D LMMs and outperforms several large proprietary 2D models. Notably, Qwen-3D achieves these improvements while maintaining strong performance on standard 2D vision-language benchmarks by jointly training on 2D and 3D data.

Related papers

• 3D-Aware VLMs with Implicit and Explicit Geometries [72.8641426724502]
  We present VLM-IE3D, a unified framework that enhances the 3D spatial awareness of vision-language models (VLMs)<n>Our VLM-IE3D introduces Implicit Geometry Tokens (IGTs) that capture high-level geometric priors from input videos, as well as complementary Explicit Geometry Tokens (EGTs) that encode detailed geometric structures from reconstructed 3D attributes.<n>VLM-IE3D achieves superior performance consistently across various 3D tasks including 3D video detection, 3D visual grounding, 3D dense captioning, and spatial reasoning.
  arXiv  Detail & Related papers  (2026-07-23T17:59:59Z)
• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding [31.76161781831759]
  We propose Occ-VLM, a novel framework for 3D scene understanding that operates purely on posed RGB images and employs a single 2D vision encoder.<n>Occ-VLM achieves both accurate geometric perception and robust vision-language reasoning.
  arXiv  Detail & Related papers  (2026-06-18T04:24:28Z)
• Geometry-Guided 3D Visual Token Pruning for Video-Language Models [35.786221225043455]
  We propose Geo3DPruner, a Geometry-Guided 3D Visual Token Pruning framework.<n>Geo3DPruner first models cross-frame relevance through geometry-aware global attention, and then performs a two-stage pruning process.<n>Experiments on multiple 3D scene understanding benchmarks demonstrate that Geo3DPruner retains over 90% of the original performance while pruning 90% of visual tokens, significantly outperforming existing text-guided and vision-guided pruning methods.
  arXiv  Detail & Related papers  (2026-04-20T13:33:50Z)
• Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps [77.63233146945718]
  We introduce Cog3DMap, a framework that recurrently constructs an explicit 3D memory from multi-view images.<n>Our framework enables direct reasoning over a spatially structured 3D map, achieving state-of-the-art performance on various spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-24T10:05:32Z)
• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight [105.9472902251177]
  We present a VLM-native recipe that casts 3D detection as a next-token prediction problem.<n>Our model achieves state-of-the-art results, with 49.89 AP_3D, surpassing the previous best by +15.51 absolute improvement.
  arXiv  Detail & Related papers  (2025-11-25T18:59:45Z)
• 3D Aware Region Prompted Vision Language Model [99.4106711584584]
  SR-3D connects single-view 2D images and multi-view 3D data through a shared visual token space.<n> SR-3D supports flexible region prompting, allowing users to annotate regions with bounding boxes, segmentation masks on any frame, or directly in 3D.
  arXiv  Detail & Related papers  (2025-09-16T17:59:06Z)
• Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs [72.11701578308804]
  This paper categorizes recent 3D Vision-Language Models into 3D object-centric, 2D image-based, and 3D scene-centric approaches.<n>Despite the architectural similarity of 3D scene-centric VLMs to their 2D counterparts, they have exhibited comparatively lower performance compared with the latest 3D object-centric and 2D image-based approaches.<n>Our investigation suggests that while these models possess cross-modal alignment capabilities, they tend to over-rely on linguistic cues and overfit to frequent answer distributions.
  arXiv  Detail & Related papers  (2025-06-05T17:56:12Z)
• NVSMask3D: Hard Visual Prompting with Camera Pose Interpolation for 3D Open Vocabulary Instance Segmentation [14.046423852723615]
  We introduce a novel 3D Gaussian Splatting based hard visual prompting approach to generate diverse viewpoints around target objects.<n>Our method simulates realistic 3D perspectives, effectively augmenting existing hard visual prompts.<n>This training-free strategy integrates seamlessly with prior hard visual prompts, enriching object-descriptive features.
  arXiv  Detail & Related papers  (2025-04-20T14:39:27Z)
• 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark [25.311698492216127]
  3D spatial reasoning is the ability to analyze and interpret the positions, orientations, and spatial relationships of objects within the 3D space.<n>Large multi-modal models (LMMs) have achieved remarkable progress in a wide range of image and video understanding tasks.<n>We present the first comprehensive 3D spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual question-answer pairs.
  arXiv  Detail & Related papers  (2024-12-10T18:55:23Z)
• Unsupervised Learning of Visual 3D Keypoints for Control [104.92063943162896]
  Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. We propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space.
  arXiv  Detail & Related papers  (2021-06-14T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.