Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections

• URL: http://arxiv.org/abs/2409.14677v2
• Date: Sun, 26 Jan 2025 19:07:42 GMT
• Title: Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections
• Authors: Ankit Dhiman, Manan Shah, Rishubh Parihar, Yash Bhalgat, Lokesh R Boregowda, R Venkatesh Babu,
• Abstract summary: We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality, realistic, shape and appearance-aware reflections of real-world objects. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis.
• Score: 26.02117310176884
• License:
• Abstract: We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198k samples rendered from 66k unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality, realistic, shape and appearance-aware reflections of real-world objects. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion-based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike. The project page is available at: https://val.cds.iisc.ac.in/reflecting-reality.github.io/.

Related papers

• NeRFs are Mirror Detectors: Using Structural Similarity for Multi-View Mirror Scene Reconstruction with 3D Surface Primitives [7.116175288307167]
  We present NeRF-MD, a method which shows that NeRFs can be considered as mirror detectors. We first compute an initial estimate of the scene geometry by training a standard NeRF. We then jointly optimize the radiance field and mirror geometry in a second training stage to refine their quality.
  arXiv  Detail & Related papers  (2025-01-07T18:59:53Z)
• EnvGS: Modeling View-Dependent Appearance with Environment Gaussian [78.74634059559891]
  EnvGS is a novel approach that employs a set of Gaussian primitives as an explicit 3D representation for capturing reflections of environments. To efficiently render these environment Gaussian primitives, we developed a ray-tracing-based reflection that leverages the GPU's RT core for fast rendering. Results from multiple real-world and synthetic datasets demonstrate that our method produces significantly more detailed reflections.
  arXiv  Detail & Related papers  (2024-12-19T18:59:57Z)
• Gaussian Splatting in Mirrors: Reflection-Aware Rendering via Virtual Camera Optimization [14.324573496923792]
  3D-GS often misinterprets reflections as virtual spaces, resulting in blurred and inconsistent multi-view rendering within mirrors. Our paper presents a novel method aimed at obtaining high-quality multi-view consistent reflection rendering by modelling reflections as physically-based virtual cameras.
  arXiv  Detail & Related papers  (2024-10-02T14:53:24Z)
• Multi-times Monte Carlo Rendering for Inter-reflection Reconstruction [51.911195773164245]
  Inverse rendering methods have achieved remarkable performance in reconstructing high-fidelity 3D objects with disentangled geometries, materials, and environmental light. We propose Ref-MC2 that introduces the multi-time Monte Carlo sampling which comprehensively computes the environmental illumination. We also show downstream applications, e.g., relighting and material editing, to illustrate the disentanglement ability of our method.
  arXiv  Detail & Related papers  (2024-07-08T09:27:34Z)
• MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections [58.003014868772254]
  MirrorGaussian is the first method for mirror scene reconstruction with real-time rendering based on 3D Gaussian Splatting. We introduce an intuitive dual-rendering strategy that enables differentiableization of both the real-world 3D Gaussians and the mirrored counterpart. Our approach significantly outperforms existing methods, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2024-05-20T09:58:03Z)
• Mirror-NeRF: Learning Neural Radiance Fields for Mirrors with Whitted-Style Ray Tracing [33.852910220413655]
  We present a novel neural rendering framework, named Mirror-NeRF, which is able to learn accurate geometry and reflection of the mirror. Mirror-NeRF supports various scene manipulation applications with mirrors, such as adding new objects or mirrors into the scene and synthesizing the reflections of these new objects in mirrors.
  arXiv  Detail & Related papers  (2023-08-07T03:48:07Z)
• NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images [44.1333444097976]
  We present a neural rendering-based method called NeRO for reconstructing the geometry and the BRDF of reflective objects from multiview images captured in an unknown environment.
  arXiv  Detail & Related papers  (2023-05-27T07:40:07Z)
• Relightify: Relightable 3D Faces from a Single Image via Diffusion Models [86.3927548091627]
  We present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent estimation.
  arXiv  Detail & Related papers  (2023-05-10T11:57:49Z)
• IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes [99.76677232870192]
  We show how a dense vision transformer, IRISformer, excels at both single-task and multi-task reasoning required for inverse rendering. Specifically, we propose a transformer architecture to simultaneously estimate depths, normals, spatially-varying albedo, roughness and lighting from a single image of an indoor scene. Our evaluations on benchmark datasets demonstrate state-of-the-art results on each of the above tasks, enabling applications like object insertion and material editing in a single unconstrained real image.
  arXiv  Detail & Related papers  (2022-06-16T19:50:55Z)
• Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images [59.906948203578544]
  We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatially-varying BRDF of an arbitrary object. We first estimate per-view depth maps using a deep multi-view stereo network. These depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture.
  arXiv  Detail & Related papers  (2020-03-27T21:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.