GenAssets: Generating in-the-wild 3D Assets in Latent Space

• URL: http://arxiv.org/abs/2604.23010v1
• Date: Fri, 24 Apr 2026 20:56:55 GMT
• Title: GenAssets: Generating in-the-wild 3D Assets in Latent Space
• Abstract summary: High-quality 3D assets for traffic participants are critical for multi-sensor simulation.<n>Building assets from in-the-wild data is key for diversity and realism.<n>We propose a 3D latent diffusion model that learns on in-the-wild LiDAR and camera data.
• Score: 31.627626425629035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-quality 3D assets for traffic participants are critical for multi-sensor simulation, which is essential for the safe end-to-end development of autonomy. Building assets from in-the-wild data is key for diversity and realism, but existing neural-rendering based reconstruction methods are slow and generate assets that render well only from viewpoints close to the original observations, limiting their usefulness in simulation. Recent diffusion-based generative models build complete and diverse assets, but perform poorly on in-the-wild driving scenes, where observed actors are captured under sparse and limited fields of view, and are partially occluded. In this work, we propose a 3D latent diffusion model that learns on in-the-wild LiDAR and camera data captured by a sensor platform and generates high-quality 3D assets with complete geometry and appearance. Key to our method is a "reconstruct-then-generate" approach that first leverages occlusion-aware neural rendering trained over multiple scenes to build a high-quality latent space for objects, and then trains a diffusion model that operates on the latent space. We show our method outperforms existing reconstruction and generation based methods, unlocking diverse and scalable content creation for simulation.

Related papers

• Sparse auto-regressive modeling for scene generation from multi-view images [20.272184280044083]
  We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion.<n>Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space.<n>We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work.
  arXiv  Detail & Related papers  (2026-09-03T14:41:34Z)
• Asset Harvester: Extracting 3D Assets from Autonomous Driving Logs for Simulation [63.01022057888141]
  Asset Harvester is an image-to-3D model and end-to-end pipeline that converts sparse, in-thewild object observations from real driving logs into complete, simulation-ready assets.<n>SparseViewDiT is explicitly designed to address limited-angle views and other real-world data challenges.
  arXiv  Detail & Related papers  (2026-04-20T16:20:57Z)
• SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latents [55.26273954022457]
  Estimating the material property field of 3D assets is critical for physics-based simulation, robotics, and digital twin generation.<n>We present SLAT-Phys, an end-to-end method that predicts spatially varying material property fields of 3D assets directly from a single RGB image without explicit 3D reconstruction.
  arXiv  Detail & Related papers  (2026-03-25T06:14:03Z)
• SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis [3.614325475261039]
  3D asset insertion and novel view synthesis (NVS) are key components for autonomous driving simulation, enhancing the diversity of training data.<n>We present SCPainter, a unified framework which integrates 3D Splat (GS) car asset representations and 3D scene point clouds with diffusion-based generation.<n>The 3D GS assets and 3D scene point clouds are projected together into novel views, and these projections are used to condition a diffusion model to generate high quality images.
  arXiv  Detail & Related papers  (2025-12-27T21:28:48Z)
• R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation [78.26308457952636]
  This paper introduces R3D2, a lightweight, one-step diffusion model designed to overcome limitations in autonomous driving simulation.<n>It enables realistic insertion of complete 3D assets into existing scenes by generating plausible rendering effects-such as shadows and consistent lighting-in real time.<n>We show that R3D2 significantly enhances the realism of inserted assets, enabling use-cases like text-to-3D asset insertion and cross-scene/dataset object transfer.
  arXiv  Detail & Related papers  (2025-06-09T14:50:19Z)
• DreamPolish: Domain Score Distillation With Progressive Geometry Generation [66.94803919328815]
  We introduce DreamPolish, a text-to-3D generation model that excels in producing refined geometry and high-quality textures. In the geometry construction phase, our approach leverages multiple neural representations to enhance the stability of the synthesis process. In the texture generation phase, we introduce a novel score distillation objective, namely domain score distillation (DSD), to guide neural representations toward such a domain.
  arXiv  Detail & Related papers  (2024-11-03T15:15:01Z)
• Breathing New Life into 3D Assets with Generative Repainting [74.80184575267106]
  Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, and orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools.
  arXiv  Detail & Related papers  (2023-09-15T16:34:51Z)
• AutoDecoding Latent 3D Diffusion Models [95.7279510847827]
  We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations.
  arXiv  Detail & Related papers  (2023-07-07T17:59:14Z)
• GINA-3D: Learning to Generate Implicit Neural Assets in the Wild [38.51391650845503]
  GINA-3D is a generative model that uses real-world driving data from camera and LiDAR sensors to create 3D implicit neural assets of diverse vehicles and pedestrians. We construct a large-scale object-centric dataset containing over 1.2M images of vehicles and pedestrians. We demonstrate that it achieves state-of-the-art performance in quality and diversity for both generated images and geometries.
  arXiv  Detail & Related papers  (2023-04-04T23:41:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.