FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from
Sparse Views
- URL: http://arxiv.org/abs/2303.14368v1
- Date: Sat, 25 Mar 2023 05:47:08 GMT
- Title: FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from
Sparse Views
- Authors: Vinoj Jayasundara, Amit Agrawal, Nicolas Heron, Abhinav Shrivastava,
Larry S. Davis
- Abstract summary: We present FlexNeRF, a method for photorealistic freeviewpoint rendering of humans in motion from monocular videos.
Our approach works well with sparse views, which is a challenging scenario when the subject is exhibiting fast/complex motions.
Thanks to our novel temporal and cyclic consistency constraints, our approach provides high quality outputs as the observed views become sparser.
- Score: 71.77680030806513
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We present FlexNeRF, a method for photorealistic freeviewpoint rendering of
humans in motion from monocular videos. Our approach works well with sparse
views, which is a challenging scenario when the subject is exhibiting
fast/complex motions. We propose a novel approach which jointly optimizes a
canonical time and pose configuration, with a pose-dependent motion field and
pose-independent temporal deformations complementing each other. Thanks to our
novel temporal and cyclic consistency constraints along with additional losses
on intermediate representation such as segmentation, our approach provides high
quality outputs as the observed views become sparser. We empirically
demonstrate that our method significantly outperforms the state-of-the-art on
public benchmark datasets as well as a self-captured fashion dataset. The
project page is available at: https://flex-nerf.github.io/
Related papers
- D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures.
Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
arXiv Detail & Related papers (2024-06-14T14:35:44Z) - Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling [70.34875558830241]
We present a way for learning a-temporal (4D) embedding, based on semantic semantic gears to allow for stratified modeling of dynamic regions of rendering the scene.
At the same time, almost for free, our tracking approach enables free-viewpoint of interest - a functionality not yet achieved by existing NeRF-based methods.
arXiv Detail & Related papers (2024-06-06T03:37:39Z) - Explorative Inbetweening of Time and Space [46.77750028273578]
We introduce bounded generation to control video generation based only on a given start and end frame.
Time Reversal Fusion fuses the temporally forward and backward denoising paths conditioned on the start and end frame.
We find that Time Reversal Fusion outperforms related work on all subtasks.
arXiv Detail & Related papers (2024-03-21T17:57:31Z) - DyBluRF: Dynamic Neural Radiance Fields from Blurry Monocular Video [18.424138608823267]
We propose DyBluRF, a dynamic radiance field approach that synthesizes sharp novel views from a monocular video affected by motion blur.
To account for motion blur in input images, we simultaneously capture the camera trajectory and object Discrete Cosine Transform (DCT) trajectories within the scene.
arXiv Detail & Related papers (2024-03-15T08:48:37Z) - CTNeRF: Cross-Time Transformer for Dynamic Neural Radiance Field from Monocular Video [25.551944406980297]
We propose a novel approach to generate high-quality novel views from monocular videos of complex and dynamic scenes.
We introduce a module that operates in both the time and frequency domains to aggregate the features of object motion.
Our experiments demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
arXiv Detail & Related papers (2024-01-10T00:40:05Z) - Fast View Synthesis of Casual Videos with Soup-of-Planes [24.35962788109883]
Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax.
This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently.
Our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.
arXiv Detail & Related papers (2023-12-04T18:55:48Z) - SPARF: Neural Radiance Fields from Sparse and Noisy Poses [58.528358231885846]
We introduce Sparse Pose Adjusting Radiance Field (SPARF) to address the challenge of novel-view synthesis.
Our approach exploits multi-view geometry constraints in order to jointly learn the NeRF and refine the camera poses.
arXiv Detail & Related papers (2022-11-21T18:57:47Z) - DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene.
We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views.
We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
arXiv Detail & Related papers (2022-11-20T20:57:02Z) - DeFMO: Deblurring and Shape Recovery of Fast Moving Objects [139.67524021201103]
generative model embeds an image of the blurred object into a latent space representation, disentangles the background, and renders the sharp appearance.
DeFMO outperforms the state of the art and generates high-quality temporal super-resolution frames.
arXiv Detail & Related papers (2020-12-01T16:02:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.