xModel-KD: Cross-modal Knowledge Distillation for 3D Scene Perception using LiDAR

• URL: http://arxiv.org/abs/2605.30111v1
• Date: Thu, 28 May 2026 15:48:38 GMT
• Title: xModel-KD: Cross-modal Knowledge Distillation for 3D Scene Perception using LiDAR
• Abstract summary: We propose a novel cross-modal knowledge distillation framework, xModel-KD, for 3D point cloud segmentation.<n>Our method exploits the complementary strengths of 2D texture and 3D geometry by learning unified per-point representations.<n> Experimental results show that cross-modal fusion achieves a 2% absolute improvement in mIoU over a LiDAR-only baseline.
• Score: 1.0055428846517074
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Point cloud segmentation is a fundamental task in 3D scene understanding. Its progress is constrained by the high cost and time required for dense 3D annotations, making labeled samples difficult to obtain. Beyond annotation scarcity, different sensing modalities face inherent limitations. 2D images provide rich texture and appearance cues, yet they lack explicit depth and geometric structure. In contrast, 3D point clouds capture accurate spatial geometry but are sparse and contain no texture information. As a result, relying on a single modality restricts the richness of learned representations and weakens generalization. Although recent multi-modal methods that combine 3D point clouds with 2D images have demonstrated strong performance in tasks such as classification and retrieval, they typically depend on large-scale labeled datasets and have not been fully exploited for data-efficient dense prediction. To address these limitations, we propose a novel cross-modal knowledge distillation framework, xModel-KD, for 3D point cloud segmentation. Our method exploits the complementary strengths of 2D texture and 3D geometry by learning unified per-point representations through cross-modal alignment. Specifically, we design a cross-modal fusion encoder trained with a contrastive objective that enforces feature consistency between corresponding 2D and 3D representations across multiple views. By integrating powerful pre-trained backbones with a targeted fusion strategy, the proposed framework effectively transfers appearance cues from images to geometry-aware point features. Experimental results show that cross-modal fusion achieves a 2% absolute improvement in mIoU over a LiDAR-only baseline, demonstrating the benefit of leveraging complementary multi-modal information for scalable and annotation-efficient 3D scene understanding.

Related papers

• CMF-IoU: Multi-Stage Cross-Modal Fusion 3D Object Detection with IoU Joint Prediction [29.7092783661859]
  Multi-modal methods based on camera and LiDAR sensors have garnered significant attention in the field of 3D detection.<n>We introduce a multi-stage cross-modal fusion 3D detection framework, termed CMF-IOU, to address the challenge of aligning 3D spatial and 2D semantic information.
  arXiv  Detail & Related papers  (2025-08-18T13:32:07Z)
• TriCLIP-3D: A Unified Parameter-Efficient Framework for Tri-Modal 3D Visual Grounding based on CLIP [52.79100775328595]
  3D visual grounding allows an embodied agent to understand visual information in real-world 3D environments based on human instructions.<n>Existing 3D visual grounding methods rely on separate encoders for different modalities.<n>We propose a unified 2D pre-trained multi-modal network to process all three modalities.
  arXiv  Detail & Related papers  (2025-07-20T10:28:06Z)
• Enhanced Cross-modal 3D Retrieval via Tri-modal Reconstruction [4.820576346277399]
  Cross-modal 3D retrieval is a critical yet challenging task, aiming to achieve bi-directional retrieval between 3D and text modalities.<n>We propose to adopt multi-view images and point clouds to jointly represent 3D shapes, facilitating tri-modal alignment.<n>Our method significantly outperforms previous state-of-the-art methods in both shape-to-text and text-to-shape retrieval tasks.
  arXiv  Detail & Related papers  (2025-04-02T08:29:42Z)
• DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields [68.94868475824575]
  This paper introduces a novel approach capable of generating infinite, high-quality 3D-consistent 2D annotations alongside 3D point cloud segmentations. We leverage the strong semantic prior within a 3D generative model to train a semantic decoder. Once trained, the decoder efficiently generalizes across the latent space, enabling the generation of infinite data.
  arXiv  Detail & Related papers  (2023-11-18T21:58:28Z)
• Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation [23.110443633049382]
  We propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds. Our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on Semantic KITTI by using the knowledge of KITTI360 and GTA5.
  arXiv  Detail & Related papers  (2023-09-19T14:29:57Z)
• Intrinsic Image Decomposition Using Point Cloud Representation [13.771632868567277]
  We introduce Point Intrinsic Net (PoInt-Net), which leverages 3D point cloud data to concurrently estimate albedo and shading maps. PoInt-Net is efficient, achieving consistent performance across point clouds of any size with training only required on small-scale point clouds.
  arXiv  Detail & Related papers  (2023-07-20T14:51:28Z)
• Joint-MAE: 2D-3D Joint Masked Autoencoders for 3D Point Cloud Pre-training [65.75399500494343]
  Masked Autoencoders (MAE) have shown promising performance in self-supervised learning for 2D and 3D computer vision. We propose Joint-MAE, a 2D-3D joint MAE framework for self-supervised 3D point cloud pre-training.
  arXiv  Detail & Related papers  (2023-02-27T17:56:18Z)
• PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition [55.38462937452363]
  We propose a unified multi-view cross-modal distillation architecture, including a pretrained deep image encoder as the teacher and a deep point encoder as the student. By pair-wise aligning multi-view visual and geometric descriptors, we can obtain more powerful deep point encoders without exhausting and complicated network modification.
  arXiv  Detail & Related papers  (2022-07-07T07:23:20Z)
• Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR-based Perception [122.53774221136193]
  State-of-the-art methods for driving-scene LiDAR-based perception often project the point clouds to 2D space and then process them via 2D convolution. A natural remedy is to utilize the 3D voxelization and 3D convolution network. We propose a new framework for the outdoor LiDAR segmentation, where cylindrical partition and asymmetrical 3D convolution networks are designed to explore the 3D geometric pattern.
  arXiv  Detail & Related papers  (2021-09-12T06:25:11Z)
• Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation [87.54570024320354]
  State-of-the-art methods for large-scale driving-scene LiDAR semantic segmentation often project and process the point clouds in the 2D space. A straightforward solution to tackle the issue of 3D-to-2D projection is to keep the 3D representation and process the points in the 3D space. We develop a 3D cylinder partition and a 3D cylinder convolution based framework, termed as Cylinder3D, which exploits the 3D topology relations and structures of driving-scene point clouds.
  arXiv  Detail & Related papers  (2020-08-04T13:56:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.