RefinedFields: Radiance Fields Refinement for Unconstrained Scenes

• URL: http://arxiv.org/abs/2312.00639v3
• Date: Fri, 19 Apr 2024 14:14:59 GMT
• Title: RefinedFields: Radiance Fields Refinement for Unconstrained Scenes
• Authors: Karim Kassab, Antoine Schnepf, Jean-Yves Franceschi, Laurent Caraffa, Jeremie Mary, Valérie Gouet-Brunet,
• Abstract summary: We propose RefinedFields, to the best of our knowledge, the first method leveraging pre-trained models to improve in-the-wild scene modeling. We employ pre-trained networks to refine K-Planes representations via optimization guidance. We carry out extensive experiments and verify the merit of our method on synthetic data and real tourism photo collections.
• Score: 7.421845364041002
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modeling large scenes from unconstrained images has proven to be a major challenge in computer vision. Existing methods tackling in-the-wild scene modeling operate in closed-world settings, where no conditioning on priors acquired from real-world images is present. We propose RefinedFields, which is, to the best of our knowledge, the first method leveraging pre-trained models to improve in-the-wild scene modeling. We employ pre-trained networks to refine K-Planes representations via optimization guidance using an alternating training procedure. We carry out extensive experiments and verify the merit of our method on synthetic data and real tourism photo collections. RefinedFields enhances rendered scenes with richer details and improves upon its base representation on the task of novel view synthesis in the wild. Our project page can be found at https://refinedfields.github.io.

Related papers

• PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference [62.72779589895124]
  We make the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework. We train a reward model with a dataset we construct, consisting of nearly 51,000 images annotated with human preferences. Experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-10-29T11:49:39Z)
• Reconstructing Training Data From Real World Models Trained with Transfer Learning [29.028185455223785]
  We present a novel approach enabling data reconstruction in realistic settings for models trained on high-resolution images. Our method adapts the reconstruction scheme of arXiv:2206.07758 to real-world scenarios. We introduce a novel clustering-based method to identify good reconstructions from thousands of candidates.
  arXiv  Detail & Related papers  (2024-07-22T17:59:10Z)
• TensoIR: Tensorial Inverse Rendering [51.57268311847087]
  TensoIR is a novel inverse rendering approach based on tensor factorization and neural fields. TensoRF is a state-of-the-art approach for radiance field modeling.
  arXiv  Detail & Related papers  (2023-04-24T21:39:13Z)
• Progressively Optimized Local Radiance Fields for Robust View Synthesis [76.55036080270347]
  We present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window.
  arXiv  Detail & Related papers  (2023-03-24T04:03:55Z)
• Neural Radiance Transfer Fields for Relightable Novel-view Synthesis with Global Illumination [63.992213016011235]
  We propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art.
  arXiv  Detail & Related papers  (2022-07-27T16:07:48Z)
• ERF: Explicit Radiance Field Reconstruction From Scratch [12.254150867994163]
  We propose a novel explicit dense 3D reconstruction approach that processes a set of images of a scene with sensor poses and calibrations and estimates a photo-real digital model. One of the key innovations is that the underlying volumetric representation is completely explicit. We show that our method is general and practical. It does not require a highly controlled lab setup for capturing, but allows for reconstructing scenes with a vast variety of objects.
  arXiv  Detail & Related papers  (2022-02-28T19:37:12Z)
• A Generative Model for Generic Light Field Reconstruction [15.394019131959096]
  We present for the first time a generative model for 4D light field patches using variational autoencoders. We develop a generative model conditioned on the central view of the light field and incorporate this as a prior in an energy minimization framework. Our proposed method demonstrates good reconstruction, with performance approaching end-to-end trained networks.
  arXiv  Detail & Related papers  (2020-05-13T18:27:42Z)
• Two-shot Spatially-varying BRDF and Shape Estimation [89.29020624201708]
  We propose a novel deep learning architecture with a stage-wise estimation of shape and SVBRDF. We create a large-scale synthetic training dataset with domain-randomized geometry and realistic materials. Experiments on both synthetic and real-world datasets show that our network trained on a synthetic dataset can generalize well to real-world images.
  arXiv  Detail & Related papers  (2020-04-01T12:56:13Z)
• Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement [78.58603635621591]
  Training an unpaired synthetic-to-real translation network in image space is severely under-constrained. We propose a semi-supervised approach that operates on the disentangled shading and albedo layers of the image. Our two-stage pipeline first learns to predict accurate shading in a supervised fashion using physically-based renderings as targets.
  arXiv  Detail & Related papers  (2020-03-27T21:45:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.