Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning

• URL: http://arxiv.org/abs/2605.13852v1
• Date: Wed, 25 Mar 2026 11:30:22 GMT
• Title: Realiz3D: 3D Generation Made Photorealistic via Domain-Aware Learning
• Abstract summary: Realiz3D is a framework to generate 3D-based images that are both photorealistic and 3D-consistent.<n>We show how Realiz3D mitigates the unintended presence of control signals achieved by fine tuning an image generator.<n>We also show how the model can be guided to produce realistic images even when controls are applied.
• Score: 60.27167498639778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We often aim to generate images that are both photorealistic and 3D-consistent, adhering to precise geometry, material, and viewpoint controls. Typically, this is achieved by fine-tuning an image generator, pre-trained on billions of real images, using renders of synthetic 3D assets, where annotations for control signals are available. While this approach can learn the desired controls, it often compromises the realism of the images due to domain gap between photographs and renders. We observe that this issue largely arises from the model learning an unintended association between the presence of control signals and the synthetic appearance of the images. To address this, we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain. The key idea is to explicitly learn visual domain, real or synthetic, separately from other control signals by introducing a co-variate that, fed into small residual adapters, shifts the domain. Then, the generator can be trained to gain controllability, without fitting to specific visual domain. In this way, the model can be guided to produce realistic images even when controls are applied. We enhance control transferability to the real domain by leveraging insights about roles of different layers and denoising steps in diffusion-based generators, informing new training and inference strategies that further mitigate the gap. We demonstrate the advantages of Realiz3D in tasks as text-to-multiview generation and texturing from 3D inputs, producing outputs that are 3D-consistent and photorealistic.

Related papers

• ViewMorpher3D: A 3D-aware Diffusion Framework for Multi-Camera Novel View Synthesis in Autonomous Driving [20.935790354765604]
  We introduce ViewMorpher3D, a multi-view image enhancement framework based on image diffusion models.<n>Unlike single-view approaches, ViewMorpher3D jointly processes a set of rendered views conditioned on camera poses, 3D geometric priors, and temporally adjacent or spatially overlapping reference views.<n>Our framework accommodates variable numbers of cameras and flexible reference/target view configurations, making it adaptable to diverse sensor setups.
  arXiv  Detail & Related papers  (2026-01-12T13:44:14Z)
• Real3D: Scaling Up Large Reconstruction Models with Real-World Images [34.735198125706326]
  Real3D is the first LRM system that can be trained using single-view real-world images. We propose two unsupervised losses that allow us to supervise LRMs at the pixel- and semantic-level. We develop an automatic data curation approach to collect high-quality examples from in-the-wild images.
  arXiv  Detail & Related papers  (2024-06-12T17:59:08Z)
• Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance Fields using Geometry-Guided Text-to-Image Diffusion Model [39.64952340472541]
  We propose a controllable text-to-3D avatar generation method whose facial expression is controllable. Our main strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF) optimized with a set of controlled viewpoint-aware images. We demonstrate the empirical results and discuss the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-09-07T08:14:46Z)
• IT3D: Improved Text-to-3D Generation with Explicit View Synthesis [71.68595192524843]
  This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data.
  arXiv  Detail & Related papers  (2023-08-22T14:39:17Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations [89.1388369229542]
  We propose an unsupervised method for 3D geometry-aware representation learning of articulated objects. We obviate this need by learning the representations with GAN training. Experiments demonstrate the efficiency of our method and show that GAN-based training enables learning of controllable 3D representations without supervision.
  arXiv  Detail & Related papers  (2022-04-19T12:10:18Z)
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering [101.56891506498755]
  Differentiable rendering has paved the way to training neural networks to perform "inverse graphics" tasks. We show that our approach significantly outperforms state-of-the-art inverse graphics networks trained on existing datasets.
  arXiv  Detail & Related papers  (2020-10-18T22:29:07Z)
• Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition [67.9464567157846]
  We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-06-29T12:53:58Z)
• Learning Neural Light Transport [28.9247002210861]
  We present an approach for learning light transport in static and dynamic 3D scenes using a neural network. We find that our model is able to produce photorealistic renderings of static and dynamic scenes.
  arXiv  Detail & Related papers  (2020-06-05T13:26:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.