PKU-DyMVHumans: A Multi-View Video Benchmark for High-Fidelity Dynamic Human Modeling

• URL: http://arxiv.org/abs/2403.16080v3
• Date: Tue, 2 Apr 2024 10:16:05 GMT
• Title: PKU-DyMVHumans: A Multi-View Video Benchmark for High-Fidelity Dynamic Human Modeling
• Authors: Xiaoyun Zheng, Liwei Liao, Xufeng Li, Jianbo Jiao, Rongjie Wang, Feng Gao, Shiqi Wang, Ronggang Wang,
• Abstract summary: We present PKU-DyMVHumans, a versatile human-centric dataset for high-fidelity reconstruction and rendering of dynamic human scenarios. It comprises 8.2 million frames captured by more than 56 cameras synchronized across diverse scenarios. Inspired by recent advancements in neural field (NeRF)-based scene representations, we carefully set up an off-the-shelf framework.
• Score: 33.00658723633997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: High-quality human reconstruction and photo-realistic rendering of a dynamic scene is a long-standing problem in computer vision and graphics. Despite considerable efforts invested in developing various capture systems and reconstruction algorithms, recent advancements still struggle with loose or oversized clothing and overly complex poses. In part, this is due to the challenges of acquiring high-quality human datasets. To facilitate the development of these fields, in this paper, we present PKU-DyMVHumans, a versatile human-centric dataset for high-fidelity reconstruction and rendering of dynamic human scenarios from dense multi-view videos. It comprises 8.2 million frames captured by more than 56 synchronized cameras across diverse scenarios. These sequences comprise 32 human subjects across 45 different scenarios, each with a high-detailed appearance and realistic human motion. Inspired by recent advancements in neural radiance field (NeRF)-based scene representations, we carefully set up an off-the-shelf framework that is easy to provide those state-of-the-art NeRF-based implementations and benchmark on PKU-DyMVHumans dataset. It is paving the way for various applications like fine-grained foreground/background decomposition, high-quality human reconstruction and photo-realistic novel view synthesis of a dynamic scene. Extensive studies are performed on the benchmark, demonstrating new observations and challenges that emerge from using such high-fidelity dynamic data.

Related papers

• EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting [95.44545809256473]
  EgoGaussian is a method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We show significant improvements in terms of both dynamic object and background reconstruction quality compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-06-28T10:39:36Z)
• Scaling Up Dynamic Human-Scene Interaction Modeling [58.032368564071895]
  TRUMANS is the most comprehensive motion-captured HSI dataset currently available. It intricately captures whole-body human motions and part-level object dynamics. We devise a diffusion-based autoregressive model that efficiently generates HSI sequences of any length.
  arXiv  Detail & Related papers  (2024-03-13T15:45:04Z)
• LiveHPS: LiDAR-based Scene-level Human Pose and Shape Estimation in Free Environment [59.320414108383055]
  We present LiveHPS, a novel single-LiDAR-based approach for scene-level human pose and shape estimation. We propose a huge human motion dataset, named FreeMotion, which is collected in various scenarios with diverse human poses.
  arXiv  Detail & Related papers  (2024-02-27T03:08:44Z)
• HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion [114.15397904945185]
  We propose a unified framework, HyperHuman, that generates in-the-wild human images of high realism and diverse layouts. Our model enforces the joint learning of image appearance, spatial relationship, and geometry in a unified network. Our framework yields the state-of-the-art performance, generating hyper-realistic human images under diverse scenarios.
  arXiv  Detail & Related papers  (2023-10-12T17:59:34Z)
• DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering [126.00165445599764]
  We present DNA-Rendering, a large-scale, high-fidelity repository of human performance data for neural actor rendering. Our dataset contains over 1500 human subjects, 5000 motion sequences, and 67.5M frames' data volume. We construct a professional multi-view system to capture data, which contains 60 synchronous cameras with max 4096 x 3000 resolution, 15 fps speed, and stern camera calibration steps.
  arXiv  Detail & Related papers  (2023-07-19T17:58:03Z)
• HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion [7.592039690054564]
  We introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input. Our novel representation acts as a dynamic video encoding that captures fine details at high compression rates. We demonstrate challenges that emerge from using such high-resolution data and show that our newly introduced HumanRF effectively leverages this data.
  arXiv  Detail & Related papers  (2023-05-10T17:59:55Z)
• NeRSemble: Multi-view Radiance Field Reconstruction of Human Heads [2.5999037208435705]
  We propose a new multi-view capture setup composed of 16 calibrated machine vision cameras. With our setup, we collect a new dataset of over 4700 high-resolution, high-framerate sequences of more than 220 human heads. In order to reconstruct high-fidelity human heads, we propose Dynamic Neural Radiance Fields using Hash Ensembles.
  arXiv  Detail & Related papers  (2023-05-04T17:52:18Z)
• wildNeRF: Complete view synthesis of in-the-wild dynamic scenes captured using sparse monocular data [16.7345472998388]
  We present a novel neural radiance model that is trainable in a self-supervised manner for novel-view synthesis of dynamic unstructured scenes. Our end-to-end trainable algorithm learns highly complex, real-world static scenes within seconds and dynamic scenes with both rigid and non-rigid motion within minutes.
  arXiv  Detail & Related papers  (2022-09-20T14:37:56Z)
• HSPACE: Synthetic Parametric Humans Animated in Complex Environments [67.8628917474705]
  We build a large-scale photo-realistic dataset, Human-SPACE, of animated humans placed in complex indoor and outdoor environments. We combine a hundred diverse individuals of varying ages, gender, proportions, and ethnicity, with hundreds of motions and scenes, in order to generate an initial dataset of over 1 million frames. Assets are generated automatically, at scale, and are compatible with existing real time rendering and game engines.
  arXiv  Detail & Related papers  (2021-12-23T22:27:55Z)
• Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions [35.41116017268475]
  4D reconstruction of human-object interaction is critical for immersive VR/AR experience and human activity understanding. Recent advances still fail to recover fine geometry and texture results from sparse RGB inputs, especially under challenging human-object interactions scenarios. We propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects.
  arXiv  Detail & Related papers  (2021-08-01T04:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.