Compressing Video Calls using Synthetic Talking Heads
- URL: http://arxiv.org/abs/2210.03692v1
- Date: Fri, 7 Oct 2022 16:52:40 GMT
- Title: Compressing Video Calls using Synthetic Talking Heads
- Authors: Madhav Agarwal, Anchit Gupta, Rudrabha Mukhopadhyay, Vinay P.
Namboodiri, C V Jawahar
- Abstract summary: We propose an end-to-end system for talking head video compression.
Our algorithm transmits pivot frames intermittently while the rest of the talking head video is generated by animating them.
We use a state-of-the-art face reenactment network to detect key points in the non-pivot frames and transmit them to the receiver.
- Score: 43.71577046989023
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We leverage the modern advancements in talking head generation to propose an
end-to-end system for talking head video compression. Our algorithm transmits
pivot frames intermittently while the rest of the talking head video is
generated by animating them. We use a state-of-the-art face reenactment network
to detect key points in the non-pivot frames and transmit them to the receiver.
A dense flow is then calculated to warp a pivot frame to reconstruct the
non-pivot ones. Transmitting key points instead of full frames leads to
significant compression. We propose a novel algorithm to adaptively select the
best-suited pivot frames at regular intervals to provide a smooth experience.
We also propose a frame-interpolater at the receiver's end to improve the
compression levels further. Finally, a face enhancement network improves
reconstruction quality, significantly improving several aspects like the
sharpness of the generations. We evaluate our method both qualitatively and
quantitatively on benchmark datasets and compare it with multiple compression
techniques. We release a demo video and additional information at
https://cvit.iiit.ac.in/research/projects/cvit-projects/talking-video-compression.
Related papers
- Perceptual Quality Improvement in Videoconferencing using
Keyframes-based GAN [28.773037051085318]
We propose a novel GAN-based method for compression artifacts reduction in videoconferencing.
First, we extract multi-scale features from the compressed and reference frames.
Then, our architecture combines these features in a progressive manner according to facial landmarks.
arXiv Detail & Related papers (2023-11-07T16:38:23Z) - Predictive Coding For Animation-Based Video Compression [13.161311799049978]
We propose a predictive coding scheme which uses image animation as a predictor, and codes the residual with respect to the actual target frame.
Our experiments indicate a significant gain, in excess of 70% compared to the HEVC video standard and over 30% compared to VVC.
arXiv Detail & Related papers (2023-07-09T14:40:54Z) - Towards Smooth Video Composition [59.134911550142455]
Video generation requires consistent and persistent frames with dynamic content over time.
This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs)
We show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, brings a smooth frame transition without compromising the per-frame quality.
arXiv Detail & Related papers (2022-12-14T18:54:13Z) - Advancing Learned Video Compression with In-loop Frame Prediction [177.67218448278143]
In this paper, we propose an Advanced Learned Video Compression (ALVC) approach with the in-loop frame prediction module.
The predicted frame can serve as a better reference than the previously compressed frame, and therefore it benefits the compression performance.
The experiments show the state-of-the-art performance of our ALVC approach in learned video compression.
arXiv Detail & Related papers (2022-11-13T19:53:14Z) - Leveraging Bitstream Metadata for Fast, Accurate, Generalized Compressed
Video Quality Enhancement [74.1052624663082]
We develop a deep learning architecture capable of restoring detail to compressed videos.
We show that this improves restoration accuracy compared to prior compression correction methods.
We condition our model on quantization data which is readily available in the bitstream.
arXiv Detail & Related papers (2022-01-31T18:56:04Z) - Deep Contextual Video Compression [20.301569390401102]
We propose a deep contextual video compression framework to enable a paradigm shift from predictive coding to conditional coding.
Our method can significantly outperform the previous state-of-theart (SOTA) deep video compression methods.
arXiv Detail & Related papers (2021-09-30T12:14:24Z) - Perceptual Learned Video Compression with Recurrent Conditional GAN [158.0726042755]
We propose a Perceptual Learned Video Compression (PLVC) approach with recurrent conditional generative adversarial network.
PLVC learns to compress video towards good perceptual quality at low bit-rate.
The user study further validates the outstanding perceptual performance of PLVC in comparison with the latest learned video compression approaches.
arXiv Detail & Related papers (2021-09-07T13:36:57Z) - End-to-End Learning for Video Frame Compression with Self-Attention [25.23586503813838]
We propose an end-to-end learned system for compressing video frames.
Our system learns deep embeddings of frames and encodes their difference in latent space.
In our experiments, we show that the proposed system achieves high compression rates and high objective visual quality.
arXiv Detail & Related papers (2020-04-20T12:11:08Z) - Learning for Video Compression with Hierarchical Quality and Recurrent
Enhancement [164.7489982837475]
We propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network.
In our HLVC approach, the hierarchical quality benefits the coding efficiency, since the high quality information facilitates the compression and enhancement of low quality frames at encoder and decoder sides.
arXiv Detail & Related papers (2020-03-04T09:31:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.