MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving

• URL: http://arxiv.org/abs/2606.24301v1
• Date: Tue, 23 Jun 2026 08:31:39 GMT
• Title: MM-TRELLIS: Point-Cloud Guided Multi-Modal 3D Vehicle Generation in Autonomous Driving
• Abstract summary: MM-TRELLIS is a multi-modal version of TRELLIS for in-the-wild 3D vehicle generation.<n>It integrates LiDAR and image sensors from autonomous driving datasets into native 3D generative models.<n>Our method outperforms existing methods in high-fidelity 3D vehicle generation.
• Score: 48.042009630881275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recovering realistic 3D vehicle models from autonomous driving scenes is crucial for synthesizing training data and building simulation environment. However, most existing vehicle generation methods fail to fully exploit multimodal sensors i.e. multi-view images and LiDAR point clouds) and rely on neural rendering based reconstruction, leading to low-quality mesh. Recently, native 3D generative models have made significant progress, yet they are not built for arbitrary multi-view inputs and often struggle with in-the-wild driving images. In this work, we present MM-TRELLIS, a multi-modal version of TRELLIS for in-the-wild 3D vehicle generation that integrates LiDAR and image sensors from autonomous driving datasets into native 3D generative models. Specifically, multi-view images are cycled as conditioning inputs, while LiDAR point clouds provide test-time guidance to ensure geometric accuracy and cross-view consistency. During denoising, we first align the guidance point cloud with the model priors, then enforce consistency between the generated geometry and the guidance point cloud. Finally, we introduce a voxel filtering strategy based on the opacity of 3D Gaussian Splatting to suppress floaters and produce clean meshes. Comprehensive experiments on Waymo dataset demonstrate our method outperforms existing methods in high-fidelity 3D vehicle generation. Code is available at https://github.com/HongliXiao/MM-TRELLIS.

Related papers

• 3DCarGen: Scalable 3D Car Generation via 3D-consistent Multi-view Synthesis [35.43901502343984]
  3DCarGen is a scalable single-view 3D car generation framework for real-world images.<n>Our approach achieves robust geometric consistency and reconstruction fidelity compared to existing methods.
  arXiv  Detail & Related papers  (2026-06-23T07:44:14Z)
• Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation [7.970454839582266]
  We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds.<n>Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality.<n>We also enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model.
  arXiv  Detail & Related papers  (2026-01-12T16:20:20Z)
• DSplats: 3D Generation by Denoising Splats-Based Multiview Diffusion Models [67.50989119438508]
  We introduce DSplats, a novel method that directly denoises multiview images using Gaussian-based Reconstructors to produce realistic 3D assets.<n>Our experiments demonstrate that DSplats not only produces high-quality, spatially consistent outputs, but also sets a new standard in single-image to 3D reconstruction.
  arXiv  Detail & Related papers  (2024-12-11T07:32:17Z)
• HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving [29.327572707959916]
  We propose our framework, emphHoloDrive, to jointly generate the camera images and LiDAR point clouds.<n>We employ BEV-to-Camera and Camera-to-BEV transform modules between heterogeneous generative models.<n>Our method leads to significant performance gains over SOTA methods in terms of generation metrics.
  arXiv  Detail & Related papers  (2024-12-02T11:50:35Z)
• Cross-Modal Self-Supervised Learning with Effective Contrastive Units for LiDAR Point Clouds [34.99995524090838]
  3D perception in LiDAR point clouds is crucial for a self-driving vehicle to properly act in 3D environment. There has been a growing interest in self-supervised pre-training of 3D perception models. We propose the instance-aware and similarity-balanced contrastive units that are tailored for self-driving point clouds.
  arXiv  Detail & Related papers  (2024-09-10T19:11:45Z)
• MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes [72.02827211293736]
  MagicDrive3D is a novel framework for controllable 3D street scene generation.<n>It supports multi-condition control, including road maps, 3D objects, and text descriptions.<n>It generates diverse, high-quality 3D driving scenes, supports any-view rendering, and enhances downstream tasks like BEV segmentation.
  arXiv  Detail & Related papers  (2024-05-23T12:04:51Z)
• SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior [53.52396082006044]
  Current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. We propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model.
  arXiv  Detail & Related papers  (2024-03-29T09:20:29Z)
• GINA-3D: Learning to Generate Implicit Neural Assets in the Wild [38.51391650845503]
  GINA-3D is a generative model that uses real-world driving data from camera and LiDAR sensors to create 3D implicit neural assets of diverse vehicles and pedestrians. We construct a large-scale object-centric dataset containing over 1.2M images of vehicles and pedestrians. We demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.
  arXiv  Detail & Related papers  (2023-04-04T23:41:20Z)
• HUM3DIL: Semi-supervised Multi-modal 3D Human Pose Estimation for Autonomous Driving [95.42203932627102]
  3D human pose estimation is an emerging technology, which can enable the autonomous vehicle to perceive and understand the subtle and complex behaviors of pedestrians. Our method efficiently makes use of these complementary signals, in a semi-supervised fashion and outperforms existing methods with a large margin. Specifically, we embed LiDAR points into pixel-aligned multi-modal features, which we pass through a sequence of Transformer refinement stages.
  arXiv  Detail & Related papers  (2022-12-15T11:15:14Z)
• PerMO: Perceiving More at Once from a Single Image for Autonomous Driving [76.35684439949094]
  We present a novel approach to detect, segment, and reconstruct complete textured 3D models of vehicles from a single image. Our approach combines the strengths of deep learning and the elegance of traditional techniques. We have integrated these algorithms with an autonomous driving system.
  arXiv  Detail & Related papers  (2020-07-16T05:02:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.