GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

• URL: http://arxiv.org/abs/2603.23246v1
• Date: Tue, 24 Mar 2026 14:14:21 GMT
• Title: GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models
• Abstract summary: GO-Renderer is a framework that guides video generative models to achieve high-quality object rendering under arbitrary lighting conditions.<n>It achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.
• Score: 10.712561585083222
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.

Related papers

• Ink3D: Sculpting 3D Assets with Extremely Complex Textures via Video Generative Models [53.12111209077268]
  Ink3D is a framework that bridges 3D generation with large-scale video generative models to synthesize complex textures.<n> Texturer is a neural baking module that integrates dense multi-view observations while mitigating geometry inconsistencies.
  arXiv  Detail & Related papers  (2026-07-01T17:56:39Z)
• GR3EN: Generative Relighting for 3D Environments [52.77521698841744]
  We present a method for relighting 3D reconstructions of large room-scale environments.<n>Our approach enables controllable 3D relighting of room-scale scenes by distilling the outputs of a video-to-video relighting diffusion model into a 3D reconstruction.<n>This side-steps the need to solve a difficult inverse rendering problem, and results in a flexible system that can relight 3D reconstructions of complex real-world scenes.
  arXiv  Detail & Related papers  (2026-01-22T19:10:05Z)
• 3D-RE-GEN: 3D Reconstruction of Indoor Scenes with a Generative Framework [7.4570191712029965]
  3D-RE-GEN is a compositional framework that reconstructs a single image into textured 3D objects and a background.<n>Our pipeline integrates models for asset detection, reconstruction, and placement, pushing certain models beyond their originally intended domains.<n>Unlike current methods, 3D-RE-GEN generates a comprehensive background that spatially constrains objects during optimization.
  arXiv  Detail & Related papers  (2025-12-19T11:20:52Z)
• SViM3D: Stable Video Material Diffusion for Single Image 3D Generation [48.986972061812004]
  Video diffusion models have been successfully used to reconstruct 3D objects from a single image efficiently.<n>We extend a latent video diffusion model to output spatially varying PBR parameters and surface normals jointly with each generated view based on explicit camera control.<n>This unique setup allows for relighting and generating a 3D asset using our model as neural prior.
  arXiv  Detail & Related papers  (2025-10-09T14:29:47Z)
• 3D Object Manipulation in a Single Image using Generative Models [30.241857090353864]
  We introduce textbfOMG3D, a novel framework that integrates the precise geometric control with the generative power of diffusion models.<n>Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level.<n>Remarkably, all these steps can be done using one NVIDIA 3090.
  arXiv  Detail & Related papers  (2025-01-22T15:06:30Z)
• GenLit: Reformulating Single-Image Relighting as Video Generation [42.0880277180892]
  We introduce GenLit, a framework that distills the ability of a graphics engine to perform light manipulation into a video-generation model.<n>We find that a model fine-tuned on only a small synthetic dataset generalizes to real-world scenes.
  arXiv  Detail & Related papers  (2024-12-15T15:40:40Z)
• ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance [76.7746870349809]
  We present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models. Our proposed framework emphasizes spatial alignment of objects, compared with standard score distillation sampling.
  arXiv  Detail & Related papers  (2024-03-19T03:39:43Z)
• Anything-3D: Towards Single-view Anything Reconstruction in the Wild [61.090129285205805]
  We introduce Anything-3D, a methodical framework that ingeniously combines a series of visual-language models and the Segment-Anything object segmentation model. Our approach employs a BLIP model to generate textural descriptions, utilize the Segment-Anything model for the effective extraction of objects of interest, and leverages a text-to-image diffusion model to lift object into a neural radiance field.
  arXiv  Detail & Related papers  (2023-04-19T16:39:51Z)
• 3D Scene Creation and Rendering via Rough Meshes: A Lighting Transfer Avenue [49.62477229140788]
  This paper studies how to flexibly integrate reconstructed 3D models into practical 3D modeling pipelines such as 3D scene creation and rendering. We propose a lighting transfer network (LighTNet) to bridge NFR and PBR, such that they can benefit from each other.
  arXiv  Detail & Related papers  (2022-11-27T13:31:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.