Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT

• URL: http://arxiv.org/abs/2605.09719v1
• Date: Sun, 10 May 2026 19:38:29 GMT
• Title: Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT
• Abstract summary: Large-scale 3D vision-language models (VLMs) like LLaVA-3D offer strong spatial reasoning but are difficult to deploy due to high computational costs.<n>We propose a knowledge distillation framework that transfers spatial reasoning from a 7B teacher to a 2.29B student model.<n>Our approach achieves 8.7x lower inference latency and a 3x reduction in model size while retaining 54-72% of the teacher's performance.
• Score: 5.121729740393489
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale 3D vision-language models (VLMs) like LLaVA-3D offer strong spatial reasoning but are difficult to deploy due to high computational costs. We propose a knowledge distillation framework that transfers spatial reasoning from a 7B teacher to a 2.29B student model. Our approach achieves 8.7x lower inference latency and a 3x reduction in model size while retaining 54-72% of the teacher's performance. The framework utilizes VGGT as the vision encoder and a multi-task distillation pipeline with uncertainty-aware loss weighting. To improve reasoning without chain-of-thought (CoT) data, we introduce "Hidden CoT": learnable latent tokens that serve as an internal scratchpad before answer generation. This is the first use of latent scratchpad reasoning in distilled 3D VLMs. The student model jointly performs spatial description, depth estimation, and object detection. Experiments on ScanNet and 3D-FRONT show strong spatial understanding, reaching 68-72% accuracy on proximity and contact tasks. Our framework enables efficient 3D scene QA on resource-constrained platforms.

Related papers

• Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning [24.251128353444713]
  Vision-Language Models (VLMs) often struggle with robust 3D spatial reasoning.<n>We propose GASP (GeometricAware Spatial Priors), a framework that injects fundamental geometric priors directly into the LLM's transformer layers.<n>GASP employs a small correspondence head, applied as a deep supervision signal across all layers, and is trained with a dual objective leveraging ground-scale video scenes.
  arXiv  Detail & Related papers  (2026-05-28T17:00:52Z)
• Do 3D Large Language Models Really Understand 3D Spatial Relationships? [80.64317885117704]
  3D Large-Language Models claim to understand 3D worlds, especially spatial relationships among objects.<n>We find that fine-tuning a language model on text-only question-answer pairs can perform comparably or even surpass these methods on the SQA3D benchmark without using any 3D input.<n>We introduce Real-3DQA, a more rigorous evaluation benchmark that filters out easy-to-guess questions and introduces a structured taxonomy to assess various aspects of 3D reasoning.
  arXiv  Detail & Related papers  (2026-03-06T16:04:34Z)
• Splat and Distill: Augmenting Teachers with Feed-Forward 3D Reconstruction For 3D-Aware Distillation [10.159228582199924]
  Vision Foundation Models (VFMs) have achieved remarkable success when applied to various downstream 2D tasks.<n>Despite their effectiveness, they often exhibit a critical lack of 3D awareness.<n>We introduce Splat and Distill, a framework that instills robust 3D awareness into 2D VFMs by augmenting the teacher model with a fast, feed-forward 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2026-02-05T18:59:05Z)
• DC-Scene: Data-Centric Learning for 3D Scene Understanding [11.204526527127094]
  3D scene understanding plays a fundamental role in vision applications such as robotics, autonomous driving, and augmented reality.<n>We propose DC-Scene, a data-centric framework tailored for 3D scene understanding.<n>We introduce a CLIP-driven dual-indicator quality (DIQ) filter, combining vision-language alignment scores with caption-loss perplexity, along with a curriculum scheduler.
  arXiv  Detail & Related papers  (2025-05-21T08:05:27Z)
• SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining [100.23919762298227]
  Currently, all existing methods rely on 2D or textual modalities during training or together at inference.<n>We introduce SceneSplat, to our knowledge the first large-scale 3D indoor scene understanding approach that operates on 3DGS.<n>We propose a self-supervised learning scheme that unlocks rich 3D feature learning from unlabeled scenes.
  arXiv  Detail & Related papers  (2025-03-23T12:50:25Z)
• Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation [67.36775428466045]
  We propose Geometry Guided Self-Distillation (GGSD) to learn superior 3D representations from 2D pre-trained models. Due to the advantages of 3D representation, the performance of the distilled 3D student model can significantly surpass that of the 2D teacher model.
  arXiv  Detail & Related papers  (2024-07-18T10:13:56Z)
• Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection [10.84784828447741]
  ADD is an Attention-based Depth knowledge Distillation framework with 3D-aware positional encoding. Credit to our teacher design, our framework is seamless, domain-gap free, easily implementable, and is compatible with object-wise ground-truth depth. We implement our framework on three representative monocular detectors, and we achieve state-of-the-art performance with no additional inference computational cost.
  arXiv  Detail & Related papers  (2022-11-30T06:39:25Z)
• Homography Loss for Monocular 3D Object Detection [54.04870007473932]
  A differentiable loss function, termed as Homography Loss, is proposed to achieve the goal, which exploits both 2D and 3D information. Our method yields the best performance compared with the other state-of-the-arts by a large margin on KITTI 3D datasets.
  arXiv  Detail & Related papers  (2022-04-02T03:48:03Z)
• 3D-to-2D Distillation for Indoor Scene Parsing [78.36781565047656]
  We present a new approach that enables us to leverage 3D features extracted from large-scale 3D data repository to enhance 2D features extracted from RGB images. First, we distill 3D knowledge from a pretrained 3D network to supervise a 2D network to learn simulated 3D features from 2D features during the training. Second, we design a two-stage dimension normalization scheme to calibrate the 2D and 3D features for better integration. Third, we design a semantic-aware adversarial training model to extend our framework for training with unpaired 3D data.
  arXiv  Detail & Related papers  (2021-04-06T02:22:24Z)
• Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution [34.713667358316286]
  Self-driving cars need to understand 3D scenes efficiently and accurately in order to drive safely. Existing 3D perception models are not able to recognize small instances very well due to the low-resolution voxelization and aggressive downsampling. We propose Sparse Point-Voxel Convolution (SPVConv), a lightweight 3D module that equips the vanilla Sparse Convolution with the high-resolution point-based branch.
  arXiv  Detail & Related papers  (2020-07-31T14:27:27Z)
• PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding [107.02479689909164]
  In this work, we aim at facilitating research on 3D representation learning. We measure the effect of unsupervised pre-training on a large source set of 3D scenes.
  arXiv  Detail & Related papers  (2020-07-21T17:59:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.