DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

• URL: http://arxiv.org/abs/2407.16988v2
• Date: Tue, 30 Jul 2024 01:16:47 GMT
• Title: DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction
• Authors: Xiaobiao Du, Haiyang Sun, Ming Lu, Tianqing Zhu, Xin Yu,
• Abstract summary: Self-driving industries usually employ professional artists to build exquisite 3D cars. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. We propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image.
• Score: 29.095687643972784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. \href{https://xiaobiaodu.github.io/dreamcar-project/}{Our code is available.}

Related papers

• VQA-Diff: Exploiting VQA and Diffusion for Zero-Shot Image-to-3D Vehicle Asset Generation in Autonomous Driving [25.03216574230919]
  We propose VQA-Diff, a novel framework that leverages in-the-wild vehicle images to create 3D vehicle assets for autonomous driving. VQA-Diff exploits the real-world knowledge inherited from the Large Language Model in the Visual Question Answering (VQA) model for robust zero-shot prediction. We conduct experiments on various datasets, including Pascal 3D+, to demonstrate that VQA-Diff outperforms existing state-of-the-art methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2024-07-09T03:09:55Z)
• 3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views [24.414577645896415]
  3D cars are commonly used in self-driving systems, virtual/augmented reality, and games. Existing 3D car datasets are either synthetic or low-quality, presenting a significant gap toward the high-quality real-world 3D car datasets. We propose the first large-scale 3D real car dataset, termed 3DRealCar, offering three distinctive features.
  arXiv  Detail & Related papers  (2024-06-07T12:14:27Z)
• MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes [72.02827211293736]
  We introduce MagicDrive3D, a novel pipeline for controllable 3D street scene generation. Unlike previous methods that reconstruct before training the generative models, MagicDrive3D first trains a video generation model and then reconstructs from the generated data. Our results demonstrate the framework's superior performance, showcasing its potential for autonomous driving simulation and beyond.
  arXiv  Detail & Related papers  (2024-05-23T12:04:51Z)
• Surrogate Modeling of Car Drag Coefficient with Depth and Normal Renderings [4.868319717279586]
  We propose a new two-dimensional (2D) representation of 3D shapes to verify its effectiveness in predicting 3D car drag. We construct a diverse dataset of 9,070 high-quality 3D car meshes labeled by drag coefficients. Our experiments demonstrate that our model can accurately and efficiently evaluate drag coefficients with an $R2$ value above 0.84 for various car categories.
  arXiv  Detail & Related papers  (2023-05-26T09:33:12Z)
• GINA-3D: Learning to Generate Implicit Neural Assets in the Wild [38.51391650845503]
  GINA-3D is a generative model that uses real-world driving data from camera and LiDAR sensors to create 3D implicit neural assets of diverse vehicles and pedestrians. We construct a large-scale object-centric dataset containing over 1.2M images of vehicles and pedestrians. We demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.
  arXiv  Detail & Related papers  (2023-04-04T23:41:20Z)
• 3D Data Augmentation for Driving Scenes on Camera [50.41413053812315]
  We propose a 3D data augmentation approach termed Drive-3DAug, aiming at augmenting the driving scenes on camera in the 3D space. We first utilize Neural Radiance Field (NeRF) to reconstruct the 3D models of background and foreground objects. Then, augmented driving scenes can be obtained by placing the 3D objects with adapted location and orientation at the pre-defined valid region of backgrounds.
  arXiv  Detail & Related papers  (2023-03-18T05:51:05Z)
• MobileBrick: Building LEGO for 3D Reconstruction on Mobile Devices [78.20154723650333]
  High-quality 3D ground-truth shapes are critical for 3D object reconstruction evaluation. We introduce a novel multi-view RGBD dataset captured using a mobile device. We obtain precise 3D ground-truth shape without relying on high-end 3D scanners.
  arXiv  Detail & Related papers  (2023-03-03T14:02:50Z)
• One Million Scenes for Autonomous Driving: ONCE Dataset [91.94189514073354]
  We introduce the ONCE dataset for 3D object detection in the autonomous driving scenario. The data is selected from 144 driving hours, which is 20x longer than the largest 3D autonomous driving dataset available. We reproduce and evaluate a variety of self-supervised and semi-supervised methods on the ONCE dataset.
  arXiv  Detail & Related papers  (2021-06-21T12:28:08Z)
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation [77.60050239225086]
  We propose an effective training data generation process by fitting a 3D car model with dynamic parts to vehicles in real images. Our approach is fully automatic without any human interaction. We present a multi-task network for VUS parsing and a multi-stream network for VHI parsing.
  arXiv  Detail & Related papers  (2020-12-15T03:03:38Z)
• PerMO: Perceiving More at Once from a Single Image for Autonomous Driving [76.35684439949094]
  We present a novel approach to detect, segment, and reconstruct complete textured 3D models of vehicles from a single image. Our approach combines the strengths of deep learning and the elegance of traditional techniques. We have integrated these algorithms with an autonomous driving system.
  arXiv  Detail & Related papers  (2020-07-16T05:02:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.