Super-Resolution Appearance Transfer for 4D Human Performances

• URL: http://arxiv.org/abs/2108.13739v1
• Date: Tue, 31 Aug 2021 10:53:11 GMT
• Title: Super-Resolution Appearance Transfer for 4D Human Performances
• Authors: Marco Pesavento, Marco Volino and Adrian Hilton
• Abstract summary: A common problem in the 4D reconstruction of people from multi-view video is the quality of the captured dynamic texture appearance. We propose a solution through super-resolution appearance transfer from a static high-resolution appearance capture rig.
• Score: 29.361342747786164
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A common problem in the 4D reconstruction of people from multi-view video is the quality of the captured dynamic texture appearance which depends on both the camera resolution and capture volume. Typically the requirement to frame cameras to capture the volume of a dynamic performance ($>50m^3$) results in the person occupying only a small proportion $<$ 10% of the field of view. Even with ultra high-definition 4k video acquisition this results in sampling the person at less-than standard definition 0.5k video resolution resulting in low-quality rendering. In this paper we propose a solution to this problem through super-resolution appearance transfer from a static high-resolution appearance capture rig using digital stills cameras ($> 8k$) to capture the person in a small volume ($<8m^3$). A pipeline is proposed for super-resolution appearance transfer from high-resolution static capture to dynamic video performance capture to produce super-resolution dynamic textures. This addresses two key problems: colour mapping between different camera systems; and dynamic texture map super-resolution using a learnt model. Comparative evaluation demonstrates a significant qualitative and quantitative improvement in rendering the 4D performance capture with super-resolution dynamic texture appearance. The proposed approach reproduces the high-resolution detail of the static capture whilst maintaining the appearance dynamics of the captured video.

Related papers

• VisualRWKV-HD and UHD: Advancing High-Resolution Processing for Visual Language Models [1.03590082373586]
  We present VisualRWKV-HD and VisualRWKV-UHD, two advancements in the VisualRWKV model family, specifically designed to process high-resolution visual inputs. Both models support resolutions up to 4096 x 4096 pixels, offering a more detailed and comprehensive visual processing capability.
  arXiv  Detail & Related papers  (2024-10-15T14:49:19Z)
• AIM 2024 Challenge on Efficient Video Super-Resolution for AV1 Compressed Content [56.552444900457395]
  Video super-resolution (VSR) is a critical task for enhancing low-bitrate and low-resolution videos, particularly in streaming applications. In this work, we compile different methods to address these challenges, the solutions are end-to-end real-time video super-resolution frameworks. The proposed solutions tackle video up-scaling for two applications: 540p to 4K (x4) as a general case, and 360p to 1080p (x3) more tailored towards mobile devices.
  arXiv  Detail & Related papers  (2024-09-25T18:12:19Z)
• EG4D: Explicit Generation of 4D Object without Score Distillation [105.63506584772331]
  DG4D is a novel framework that generates high-quality and consistent 4D assets without score distillation. Our framework outperforms the baselines in generation quality by a considerable margin.
  arXiv  Detail & Related papers  (2024-05-28T12:47:22Z)
• Holoported Characters: Real-time Free-viewpoint Rendering of Humans from Sparse RGB Cameras [65.54875149514274]
  We present the first approach to render highly realistic free-viewpoint videos of a human actor in general apparel. At inference, our method only requires four camera views of the moving actor and the respective 3D skeletal pose. It handles actors in wide clothing, and reproduces even fine-scale dynamic detail.
  arXiv  Detail & Related papers  (2023-12-12T16:45:52Z)
• 4K4D: Real-Time 4D View Synthesis at 4K Resolution [86.6582179227016]
  This paper targets high-fidelity and real-time view of dynamic 3D scenes at 4K resolution. We propose a 4D point cloud representation that supports hardwareization and enables unprecedented rendering speed. Our representation can be rendered at over 400 FPS on the DNA-Rendering dataset at 1080p resolution and 80 FPS on the ENeRF-Outdoor dataset at 4K resolution using an 4090 GPU.
  arXiv  Detail & Related papers  (2023-10-17T17:57:38Z)
• FuseSR: Super Resolution for Real-time Rendering through Efficient Multi-resolution Fusion [38.67110413800048]
  One of the most popular solutions is to render images at a low resolution to reduce rendering overhead. In this paper, we propose an efficient and effective super-resolution method that predicts high-quality upsampled reconstructions. Experiments show that our method is able to produce temporally consistent reconstructions in $4 times 4$ and even challenging $8 times 8$ upsampling cases at 4K resolution with real-time performance.
  arXiv  Detail & Related papers  (2023-10-15T04:01:05Z)
• HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion [7.592039690054564]
  We introduce HumanRF, a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input. Our novel representation acts as a dynamic video encoding that captures fine details at high compression rates. We demonstrate challenges that emerge from using such high-resolution data and show that our newly introduced HumanRF effectively leverages this data.
  arXiv  Detail & Related papers  (2023-05-10T17:59:55Z)
• HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling [60.90470761333465]
  We present HyperReel -- a novel 6-DoF video representation. The two core components of HyperReel are: (1) a ray-conditioned sample prediction network that enables high-fidelity, high frame rate rendering at high resolutions and (2) a compact and memory-efficient dynamic volume representation.
  arXiv  Detail & Related papers  (2023-01-05T18:59:44Z)
• Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing [71.62289021118983]
  We present an efficient baseline model ESDNet for tackling 4K moire images, wherein we build a semantic-aligned scale-aware module to address the scale variation of moire patterns. Our approach outperforms state-of-the-art methods by a large margin while being much more lightweight.
  arXiv  Detail & Related papers  (2022-07-20T14:20:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.