The Devil is in the Details: Simple Remedies for Image-to-LiDAR Representation Learning

• URL: http://arxiv.org/abs/2501.09485v1
• Date: Thu, 16 Jan 2025 11:44:29 GMT
• Title: The Devil is in the Details: Simple Remedies for Image-to-LiDAR Representation Learning
• Authors: Wonjun Jo, Kwon Byung-Ki, Kim Ji-Yeon, Hawook Jeong, Kyungdon Joo, Tae-Hyun Oh,
• Abstract summary: We focus on overlooked design choices along the spatial and temporal axes. We find that fundamental design elements, e.g., the LiDAR coordinate system, quantization according to the existing input interface, are more critical than developing loss functions.
• Score: 21.088879084249328
• License:
• Abstract: LiDAR is a crucial sensor in autonomous driving, commonly used alongside cameras. By exploiting this camera-LiDAR setup and recent advances in image representation learning, prior studies have shown the promising potential of image-to-LiDAR distillation. These prior arts focus on the designs of their own losses to effectively distill the pre-trained 2D image representations into a 3D model. However, the other parts of the designs have been surprisingly unexplored. We find that fundamental design elements, e.g., the LiDAR coordinate system, quantization according to the existing input interface, and data utilization, are more critical than developing loss functions, which have been overlooked in prior works. In this work, we show that simple fixes to these designs notably outperform existing methods by 16% in 3D semantic segmentation on the nuScenes dataset and 13% in 3D object detection on the KITTI dataset in downstream task performance. We focus on overlooked design choices along the spatial and temporal axes. Spatially, prior work has used cylindrical coordinate and voxel sizes without considering their side effects yielded with a commonly deployed sparse convolution layer input interface, leading to spatial quantization errors in 3D models. Temporally, existing work has avoided cumbersome data curation by discarding unsynced data, limiting the use to only the small portion of data that is temporally synced across sensors. We analyze these effects and propose simple solutions for each overlooked aspect.

Related papers

• Shelf-Supervised Cross-Modal Pre-Training for 3D Object Detection [52.66283064389691]
  State-of-the-art 3D object detectors are often trained on massive labeled datasets. Recent works demonstrate that self-supervised pre-training with unlabeled data can improve detection accuracy with limited labels. We propose a shelf-supervised approach for generating zero-shot 3D bounding boxes from paired RGB and LiDAR data.
  arXiv  Detail & Related papers  (2024-06-14T15:21:57Z)
• PruNeRF: Segment-Centric Dataset Pruning via 3D Spatial Consistency [33.68948881727943]
  PruNeRF is a segment-centric dataset pruning framework via 3D spatial consistency. Our experiments on benchmark datasets demonstrate that PruNeRF consistently outperforms state-of-the-art methods in robustness against distractors.
  arXiv  Detail & Related papers  (2024-06-02T16:49:05Z)
• FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained Vision-Language Models [59.13757801286343]
  Few-shot class-incremental learning aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. We introduce the FILP-3D framework with two novel components: the Redundant Feature Eliminator (RFE) for feature space misalignment and the Spatial Noise Compensator (SNC) for significant noise.
  arXiv  Detail & Related papers  (2023-12-28T14:52:07Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• Review of data types and model dimensionality for cardiac DTI SMS-related artefact removal [7.497343031315105]
  We compare the effect of several input types (magnitude images vs complex images), multiple dimensionalities (2D vs 3D operations), and multiple input types (single slice vs multi-slice) on the performance of a model trained to remove artefacts. Despite our initial intuition, our experiments show that, for a fixed number of parameters, simpler 2D real-valued models outperform their more advanced 3D or complex counterparts.
  arXiv  Detail & Related papers  (2022-09-20T07:41:24Z)
• Simple and Effective Synthesis of Indoor 3D Scenes [78.95697556834536]
  We study the problem of immersive 3D indoor scenes from one or more images. Our aim is to generate high-resolution images and videos from novel viewpoints. We propose an image-to-image GAN that maps directly from reprojections of incomplete point clouds to full high-resolution RGB-D images.
  arXiv  Detail & Related papers  (2022-04-06T17:54:46Z)
• Learning-based Point Cloud Registration for 6D Object Pose Estimation in the Real World [55.7340077183072]
  We tackle the task of estimating the 6D pose of an object from point cloud data. Recent learning-based approaches to addressing this task have shown great success on synthetic datasets. We analyze the causes of these failures, which we trace back to the difference between the feature distributions of the source and target point clouds.
  arXiv  Detail & Related papers  (2022-03-29T07:55:04Z)
• Lidar-Monocular Surface Reconstruction Using Line Segments [5.542669744873386]
  We propose to leverage common geometric features that are detected in both the LIDAR scans and image data, allowing data from the two sensors to be processed in a higher-level space. We show that our method delivers results that are comparable to a state-of-the-art LIDAR survey while not requiring highly accurate ground truth pose estimates.
  arXiv  Detail & Related papers  (2021-04-06T19:49:53Z)
• Reinforced Axial Refinement Network for Monocular 3D Object Detection [160.34246529816085]
  Monocular 3D object detection aims to extract the 3D position and properties of objects from a 2D input image. Conventional approaches sample 3D bounding boxes from the space and infer the relationship between the target object and each of them, however, the probability of effective samples is relatively small in the 3D space. We propose to start with an initial prediction and refine it gradually towards the ground truth, with only one 3d parameter changed in each step. This requires designing a policy which gets a reward after several steps, and thus we adopt reinforcement learning to optimize it.
  arXiv  Detail & Related papers  (2020-08-31T17:10:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.