Representing Animatable Avatar via Factorized Neural Fields

• URL: http://arxiv.org/abs/2406.00637v1
• Date: Sun, 2 Jun 2024 06:45:38 GMT
• Title: Representing Animatable Avatar via Factorized Neural Fields
• Authors: Chunjin Song, Zhijie Wu, Bastian Wandt, Leonid Sigal, Helge Rhodin,
• Abstract summary: Per-frame rendering results can be factorized into a pose-independent component and a corresponding pose-dependent equivalent to facilitate frame consistency. We achieve a coherent preservation of both coarse body contours across the entire input video. Our network surpasses the neural radiance fields (NeRF) based state-of-the-art methods in preserving high-frequency details and ensuring consistent body contours.
• Score: 39.95141875395775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For reconstructing high-fidelity human 3D models from monocular videos, it is crucial to maintain consistent large-scale body shapes along with finely matched subtle wrinkles. This paper explores the observation that the per-frame rendering results can be factorized into a pose-independent component and a corresponding pose-dependent equivalent to facilitate frame consistency. Pose adaptive textures can be further improved by restricting frequency bands of these two components. In detail, pose-independent outputs are expected to be low-frequency, while highfrequency information is linked to pose-dependent factors. We achieve a coherent preservation of both coarse body contours across the entire input video and finegrained texture features that are time variant with a dual-branch network with distinct frequency components. The first branch takes coordinates in canonical space as input, while the second branch additionally considers features outputted by the first branch and pose information of each frame. Our network integrates the information predicted by both branches and utilizes volume rendering to generate photo-realistic 3D human images. Through experiments, we demonstrate that our network surpasses the neural radiance fields (NeRF) based state-of-the-art methods in preserving high-frequency details and ensuring consistent body contours.

Related papers

• SkelFormer: Markerless 3D Pose and Shape Estimation using Skeletal Transformers [57.46911575980854]
  We introduce SkelFormer, a novel markerless motion capture pipeline for multi-view human pose and shape estimation. Our method first uses off-the-shelf 2D keypoint estimators, pre-trained on large-scale in-the-wild data, to obtain 3D joint positions. Next, we design a regression-based inverse-kinematic skeletal transformer that maps the joint positions to pose and shape representations from heavily noisy observations.
  arXiv  Detail & Related papers  (2024-04-19T04:51:18Z)
• Leveraging Neural Radiance Field in Descriptor Synthesis for Keypoints Scene Coordinate Regression [1.2974519529978974]
  This paper introduces a pipeline for keypoint descriptor synthesis using Neural Radiance Field (NeRF) generating novel poses and feeding them into a trained NeRF model to create new views, our approach enhances the KSCR's capabilities in data-scarce environments. The proposed system could significantly improve localization accuracy by up to 50% and cost only a fraction of time for data synthesis.
  arXiv  Detail & Related papers  (2024-03-15T13:40:37Z)
• Pose Modulated Avatars from Video [22.395774558845336]
  We develop a two-branch neural network that is adaptive and explicit in the frequency domain. The first branch is a graph neural network that models correlations among body parts locally. The second branch combines these correlation features to a set of global frequencies and then modulates the feature encoding.
  arXiv  Detail & Related papers  (2023-08-23T06:49:07Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Content-aware Warping for View Synthesis [110.54435867693203]
  We propose content-aware warping, which adaptively learns the weights for pixels of a relatively large neighborhood from their contextual information via a lightweight neural network. Based on this learnable warping module, we propose a new end-to-end learning-based framework for novel view synthesis from two source views. Experimental results on structured light field datasets with wide baselines and unstructured multi-view datasets show that the proposed method significantly outperforms state-of-the-art methods both quantitatively and visually.
  arXiv  Detail & Related papers  (2022-01-22T11:35:05Z)
• Shelf-Supervised Mesh Prediction in the Wild [54.01373263260449]
  We propose a learning-based approach to infer 3D shape and pose of object from a single image. We first infer a volumetric representation in a canonical frame, along with the camera pose. The coarse volumetric prediction is then converted to a mesh-based representation, which is further refined in the predicted camera frame.
  arXiv  Detail & Related papers  (2021-02-11T18:57:10Z)
• Emergent Properties of Foveated Perceptual Systems [3.3504365823045044]
  This work is inspired by the foveated human visual system, which has higher acuity at the center of gaze and texture-like encoding in the periphery. We introduce models consisting of a first-stage textitfixed image transform followed by a second-stage textitlearnable convolutional neural network. We find that foveation with peripheral texture-based computations yields an efficient, distinct, and robust representational format of scene information.
  arXiv  Detail & Related papers  (2020-06-14T19:34:44Z)
• Identity Enhanced Residual Image Denoising [61.75610647978973]
  We learn a fully-convolutional network model that consists of a Chain of Identity Mapping Modules and residual on the residual architecture for image denoising. The proposed network produces remarkably higher numerical accuracy and better visual image quality than the classical state-of-the-art and CNN algorithms.
  arXiv  Detail & Related papers  (2020-04-26T04:52:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.