Delving into the Frequency: Temporally Consistent Human Motion Transfer
in the Fourier Space
- URL: http://arxiv.org/abs/2209.00233v1
- Date: Thu, 1 Sep 2022 05:30:23 GMT
- Title: Delving into the Frequency: Temporally Consistent Human Motion Transfer
in the Fourier Space
- Authors: Guang Yang, Wu Liu, Xinchen Liu, Xiaoyan Gu, Juan Cao, Jintao Li
- Abstract summary: Human motion transfer refers to synthesizing photo-realistic and temporally coherent videos.
Current synthetic videos suffer from the temporal inconsistency in sequential frames that significantly degrades the video quality.
We propose a novel Frequency-based human MOtion TRansfer framework, named FreMOTR, which can effectively mitigate the spatial artifacts and the temporal inconsistency of the synthesized videos.
- Score: 34.353035276767336
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Human motion transfer refers to synthesizing photo-realistic and temporally
coherent videos that enable one person to imitate the motion of others.
However, current synthetic videos suffer from the temporal inconsistency in
sequential frames that significantly degrades the video quality, yet is far
from solved by existing methods in the pixel domain. Recently, some works on
DeepFake detection try to distinguish the natural and synthetic images in the
frequency domain because of the frequency insufficiency of image synthesizing
methods. Nonetheless, there is no work to study the temporal inconsistency of
synthetic videos from the aspects of the frequency-domain gap between natural
and synthetic videos. In this paper, we propose to delve into the frequency
space for temporally consistent human motion transfer. First of all, we make
the first comprehensive analysis of natural and synthetic videos in the
frequency domain to reveal the frequency gap in both the spatial dimension of
individual frames and the temporal dimension of the video. To close the
frequency gap between the natural and synthetic videos, we propose a novel
Frequency-based human MOtion TRansfer framework, named FreMOTR, which can
effectively mitigate the spatial artifacts and the temporal inconsistency of
the synthesized videos. FreMOTR explores two novel frequency-based
regularization modules: 1) the Frequency-domain Appearance Regularization (FAR)
to improve the appearance of the person in individual frames and 2) Temporal
Frequency Regularization (TFR) to guarantee the temporal consistency between
adjacent frames. Finally, comprehensive experiments demonstrate that the
FreMOTR not only yields superior performance in temporal consistency metrics
but also improves the frame-level visual quality of synthetic videos. In
particular, the temporal consistency metrics are improved by nearly 30% than
the state-of-the-art model.
Related papers
- UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation [53.16986875759286]
We present a UniAnimate framework to enable efficient and long-term human video generation.
We map the reference image along with the posture guidance and noise video into a common feature space.
We also propose a unified noise input that supports random noised input as well as first frame conditioned input.
arXiv Detail & Related papers (2024-06-03T10:51:10Z) - Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
We propose LEAPS, an architecture-independent method for synthesizing videos from internal models.
We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
arXiv Detail & Related papers (2023-03-17T12:55:22Z) - Learning Spatiotemporal Frequency-Transformer for Low-Quality Video
Super-Resolution [47.5883522564362]
Video Super-Resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) videos.
Existing VSR techniques usually recover HR frames by extracting textures from nearby frames with known degradation processes.
We propose a novel Frequency-Transformer (FTVSR) for handling low-quality videos that carry out self-attention in a combined space-time-frequency domain.
arXiv Detail & Related papers (2022-12-27T16:26:15Z) - Time reflection and refraction in synthetic frequency dimension [3.4649462017122996]
We show that time reflection and refraction can be observed in a two-band model centered around a non-zero reference energy.
We find that modulation at microwave frequencies is sufficient to observe time boundary effects for optical waves in synthetic frequency dimension.
arXiv Detail & Related papers (2022-09-08T02:37:41Z) - Spatial-Temporal Frequency Forgery Clue for Video Forgery Detection in
VIS and NIR Scenario [87.72258480670627]
Existing face forgery detection methods based on frequency domain find that the GAN forged images have obvious grid-like visual artifacts in the frequency spectrum compared to the real images.
This paper proposes a Cosine Transform-based Forgery Clue Augmentation Network (FCAN-DCT) to achieve a more comprehensive spatial-temporal feature representation.
arXiv Detail & Related papers (2022-07-05T09:27:53Z) - Intrinsic Temporal Regularization for High-resolution Human Video
Synthesis [59.54483950973432]
temporal consistency is crucial for extending image processing pipelines to the video domain.
We propose an effective intrinsic temporal regularization scheme, where an intrinsic confidence map is estimated via the frame generator to regulate motion estimation.
We apply our intrinsic temporal regulation to single-image generator, leading to a powerful " INTERnet" capable of generating $512times512$ resolution human action videos.
arXiv Detail & Related papers (2020-12-11T05:29:45Z) - Exploring Spatial-Temporal Multi-Frequency Analysis for High-Fidelity
and Temporal-Consistency Video Prediction [12.84409065286371]
We propose a video prediction network based on multi-level wavelet analysis to deal with spatial and temporal information in a unified manner.
Our model shows significant improvements on fidelity and temporal consistency over state-of-the-art works.
arXiv Detail & Related papers (2020-02-23T13:46:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.