SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis

• URL: http://arxiv.org/abs/2512.22706v1
• Date: Sat, 27 Dec 2025 21:28:48 GMT
• Title: SCPainter: A Unified Framework for Realistic 3D Asset Insertion and Novel View Synthesis
• Authors: Paul Dobre, Jackson Cooper, Xin Wang, Hongzhou Yang,
• Abstract summary: 3D asset insertion and novel view synthesis (NVS) are key components for autonomous driving simulation, enhancing the diversity of training data.<n>We present SCPainter, a unified framework which integrates 3D Splat (GS) car asset representations and 3D scene point clouds with diffusion-based generation.<n>The 3D GS assets and 3D scene point clouds are projected together into novel views, and these projections are used to condition a diffusion model to generate high quality images.
• Score: 3.614325475261039
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D Asset insertion and novel view synthesis (NVS) are key components for autonomous driving simulation, enhancing the diversity of training data. With better training data that is diverse and covers a wide range of situations, including long-tailed driving scenarios, autonomous driving models can become more robust and safer. This motivates a unified simulation framework that can jointly handle realistic integration of inserted 3D assets and NVS. Recent 3D asset reconstruction methods enable reconstruction of dynamic actors from video, supporting their re-insertion into simulated driving scenes. While the overall structure and appearance can be accurate, it still struggles to capture the realism of 3D assets through lighting or shadows, particularly when inserted into scenes. In parallel, recent advances in NVS methods have demonstrated promising results in synthesizing viewpoints beyond the originally recorded trajectories. However, existing approaches largely treat asset insertion and NVS capabilities in isolation. To allow for interaction with the rest of the scene and to enable more diverse creation of new scenarios for training, realistic 3D asset insertion should be combined with NVS. To address this, we present SCPainter (Street Car Painter), a unified framework which integrates 3D Gaussian Splat (GS) car asset representations and 3D scene point clouds with diffusion-based generation to jointly enable realistic 3D asset insertion and NVS. The 3D GS assets and 3D scene point clouds are projected together into novel views, and these projections are used to condition a diffusion model to generate high quality images. Evaluation on the Waymo Open Dataset demonstrate the capability of our framework to enable 3D asset insertion and NVS, facilitating the creation of diverse and realistic driving data.

Related papers

• Unified Semantic Transformer for 3D Scene Understanding [55.415468022487005]
  We introduce UNITE, a novel feed-forward neural network that unifies a diverse set of 3D semantic tasks within a single model.<n>Our model operates on unseen scenes in a fully end-to-end manner and only takes a few seconds to infer the full 3D semantic geometry.<n>We demonstrate that UNITE achieves state-of-the-art performance on several different semantic tasks and even outperforms task-specific models.
  arXiv  Detail & Related papers  (2025-12-16T12:49:35Z)
• Particulate: Feed-Forward 3D Object Articulation [89.78788418174946]
  Particulate is a feed-forward approach that, given a single static 3D mesh of an everyday object, directly infers all attributes of the underlying articulated structure.<n>We train the network end-to-end on a diverse collection of articulated 3D assets from public datasets.<n>During inference, Particulate lifts the network's feed-forward prediction to the input mesh, yielding a fully articulated 3D model in seconds.
  arXiv  Detail & Related papers  (2025-12-12T18:59:51Z)
• ArtiWorld: LLM-Driven Articulation of 3D Objects in Scenes [43.19849355456126]
  ArtiWorld is a scene-aware pipeline that localizes candidate articulable objects from textual scene descriptions.<n>At the core of this pipeline is Arti4URDF, which leverages 3D point cloud and prior knowledge of a large language model.<n>We evaluate ArtiWorld at three levels: 3D simulated objects, full 3D simulated scenes, and real-world scan scenes.
  arXiv  Detail & Related papers  (2025-11-17T04:59:21Z)
• REACT3D: Recovering Articulations for Interactive Physical 3D Scenes [96.27769519526426]
  REACT3D is a framework that converts static 3D scenes into simulation-ready interactive replicas with consistent geometry.<n>We achieve state-of-the-art performance on detection/segmentation and articulation metrics across diverse indoor scenes.
  arXiv  Detail & Related papers  (2025-10-13T12:37:59Z)
• SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass [44.087747627571716]
  3D content generation has attracted significant research interest due to its applications in VR/AR and embodied AI.<n>We present SceneGen, a novel framework that takes a scene image and corresponding object masks as input, simultaneously producing multiple 3D assets.<n>We believe this paradigm offers a novel solution for high-quality 3D content generation, potentially advancing its practical applications in downstream tasks.
  arXiv  Detail & Related papers  (2025-08-21T17:59:16Z)
• R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation [78.26308457952636]
  This paper introduces R3D2, a lightweight, one-step diffusion model designed to overcome limitations in autonomous driving simulation.<n>It enables realistic insertion of complete 3D assets into existing scenes by generating plausible rendering effects-such as shadows and consistent lighting-in real time.<n>We show that R3D2 significantly enhances the realism of inserted assets, enabling use-cases like text-to-3D asset insertion and cross-scene/dataset object transfer.
  arXiv  Detail & Related papers  (2025-06-09T14:50:19Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• MagicDrive3D: Controllable 3D Generation for Any-View Rendering in Street Scenes [72.02827211293736]
  MagicDrive3D is a novel framework for controllable 3D street scene generation.<n>It supports multi-condition control, including road maps, 3D objects, and text descriptions.<n>It generates diverse, high-quality 3D driving scenes, supports any-view rendering, and enhances downstream tasks like BEV segmentation.
  arXiv  Detail & Related papers  (2024-05-23T12:04:51Z)
• Regulating Intermediate 3D Features for Vision-Centric Autonomous Driving [26.03800936700545]
  We propose to regulate intermediate dense 3D features with the help of volume rendering. Experimental results on the Occ3D and nuScenes datasets demonstrate that Vampire facilitates fine-grained and appropriate extraction of dense 3D features.
  arXiv  Detail & Related papers  (2023-12-19T04:09:05Z)
• AutoDecoding Latent 3D Diffusion Models [95.7279510847827]
  We present a novel approach to the generation of static and articulated 3D assets that has a 3D autodecoder at its core. The 3D autodecoder framework embeds properties learned from the target dataset in the latent space. We then identify the appropriate intermediate volumetric latent space, and introduce robust normalization and de-normalization operations.
  arXiv  Detail & Related papers  (2023-07-07T17:59:14Z)
• Learning 3D Dynamic Scene Representations for Robot Manipulation [21.6131570689398]
  3D scene representation for robot manipulation should capture three key object properties: permanency, completeness, and continuity. We introduce 3D Dynamic Representation (DSR), a 3D scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics. We propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR.
  arXiv  Detail & Related papers  (2020-11-03T19:23:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.