MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References

• URL: http://arxiv.org/abs/2407.13745v1
• Date: Thu, 18 Jul 2024 17:50:03 GMT
• Title: MaRINeR: Enhancing Novel Views by Matching Rendered Images with Nearby References
• Authors: Lukas Bösiger, Mihai Dusmanu, Marc Pollefeys, Zuria Bauer,
• Abstract summary: MaRINeR is a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations.
• Score: 49.71130133080821
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rendering realistic images from 3D reconstruction is an essential task of many Computer Vision and Robotics pipelines, notably for mixed-reality applications as well as training autonomous agents in simulated environments. However, the quality of novel views heavily depends of the source reconstruction which is often imperfect due to noisy or missing geometry and appearance. Inspired by the recent success of reference-based super-resolution networks, we propose MaRINeR, a refinement method that leverages information of a nearby mapping image to improve the rendering of a target viewpoint. We first establish matches between the raw rendered image of the scene geometry from the target viewpoint and the nearby reference based on deep features, followed by hierarchical detail transfer. We show improved renderings in quantitative metrics and qualitative examples from both explicit and implicit scene representations. We further employ our method on the downstream tasks of pseudo-ground-truth validation, synthetic data enhancement and detail recovery for renderings of reduced 3D reconstructions.

Related papers

• GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinement [51.97726804507328]
  We propose a novel approach for 3D mesh reconstruction from multi-view images. Our method takes inspiration from large reconstruction models that use a transformer-based triplane generator and a Neural Radiance Field (NeRF) model trained on multi-view images.
  arXiv  Detail & Related papers  (2024-06-09T05:19:24Z)
• 2L3: Lifting Imperfect Generated 2D Images into Accurate 3D [16.66666619143761]
  Multi-view (MV) 3D reconstruction is a promising solution to fuse generated MV images into consistent 3D objects. However, the generated images usually suffer from inconsistent lighting, misaligned geometry, and sparse views, leading to poor reconstruction quality. We present a novel 3D reconstruction framework that leverages intrinsic decomposition guidance, transient-mono prior guidance, and view augmentation to cope with the three issues.
  arXiv  Detail & Related papers  (2024-01-29T02:30:31Z)
• ReconFusion: 3D Reconstruction with Diffusion Priors [104.73604630145847]
  We present ReconFusion to reconstruct real-world scenes using only a few photos. Our approach leverages a diffusion prior for novel view synthesis, trained on synthetic and multiview datasets. Our method synthesizes realistic geometry and texture in underconstrained regions while preserving the appearance of observed regions.
  arXiv  Detail & Related papers  (2023-12-05T18:59:58Z)
• A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images [15.40230841242637]
  We present a novel hierarchical representation network (HRN) to achieve accurate and detailed face reconstruction from a single image. Our framework can be extended to a multi-view fashion by considering detail consistency of different views. Our method outperforms the existing methods in both reconstruction accuracy and visual effects.
  arXiv  Detail & Related papers  (2023-02-28T09:24:36Z)
• High-fidelity 3D GAN Inversion by Pseudo-multi-view Optimization [51.878078860524795]
  We present a high-fidelity 3D generative adversarial network (GAN) inversion framework that can synthesize photo-realistic novel views. Our approach enables high-fidelity 3D rendering from a single image, which is promising for various applications of AI-generated 3D content.
  arXiv  Detail & Related papers  (2022-11-28T18:59:52Z)
• Neural 3D Reconstruction in the Wild [86.6264706256377]
  We introduce a new method that enables efficient and accurate surface reconstruction from Internet photo collections. We present a new benchmark and protocol for evaluating reconstruction performance on such in-the-wild scenes.
  arXiv  Detail & Related papers  (2022-05-25T17:59:53Z)
• Extracting Triangular 3D Models, Materials, and Lighting From Images [59.33666140713829]
  We present an efficient method for joint optimization of materials and lighting from multi-view image observations. We leverage meshes with spatially-varying materials and environment that can be deployed in any traditional graphics engine.
  arXiv  Detail & Related papers  (2021-11-24T13:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.