Shortcut-V2V: Compression Framework for Video-to-Video Translation based on Temporal Redundancy Reduction

• URL: http://arxiv.org/abs/2308.08011v2
• Date: Tue, 3 Oct 2023 22:12:08 GMT
• Title: Shortcut-V2V: Compression Framework for Video-to-Video Translation based on Temporal Redundancy Reduction
• Authors: Chaeyeon Chung, Yeojeong Park, Seunghwan Choi, Munkhsoyol Ganbat, Jaegul Choo
• Abstract summary: Shortcut-V2V is a general-purpose compression framework for video-to-video translation. We show that Shourcut-V2V achieves comparable performance compared to the original video-to-video translation model.
• Score: 32.87579824212654
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video-to-video translation aims to generate video frames of a target domain from an input video. Despite its usefulness, the existing networks require enormous computations, necessitating their model compression for wide use. While there exist compression methods that improve computational efficiency in various image/video tasks, a generally-applicable compression method for video-to-video translation has not been studied much. In response, we present Shortcut-V2V, a general-purpose compression framework for video-to-video translation. Shourcut-V2V avoids full inference for every neighboring video frame by approximating the intermediate features of a current frame from those of the previous frame. Moreover, in our framework, a newly-proposed block called AdaBD adaptively blends and deforms features of neighboring frames, which makes more accurate predictions of the intermediate features possible. We conduct quantitative and qualitative evaluations using well-known video-to-video translation models on various tasks to demonstrate the general applicability of our framework. The results show that Shourcut-V2V achieves comparable performance compared to the original video-to-video translation model while saving 3.2-5.7x computational cost and 7.8-44x memory at test time.

Related papers

• Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning [71.94122309290537]
  We propose an efficient, online approach to generate dense captions for videos. Our model uses a novel autoregressive factorized decoding architecture. Our approach shows excellent performance compared to both offline and online methods, and uses 20% less compute.
  arXiv  Detail & Related papers  (2024-11-22T02:46:44Z)
• High-Efficiency Neural Video Compression via Hierarchical Predictive Learning [27.41398149573729]
  Enhanced Deep Hierarchical Video Compression-DHVC 2.0- introduces superior compression performance and impressive complexity efficiency. Uses hierarchical predictive coding to transform each video frame into multiscale representations. Supports transmission-friendly progressive decoding, making it particularly advantageous for networked video applications in the presence of packet loss.
  arXiv  Detail & Related papers  (2024-10-03T15:40:58Z)
• VideoLLaMB: Long-context Video Understanding with Recurrent Memory Bridges [42.555895949250704]
  VideoLLaMB is a novel framework that utilizes temporal memory tokens within bridge layers to allow for the encoding of entire video sequences. SceneTilling algorithm segments videos into independent semantic units to preserve semantic integrity. In terms of efficiency, VideoLLaMB, trained on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU.
  arXiv  Detail & Related papers  (2024-09-02T08:52:58Z)
• Streaming Dense Video Captioning [85.70265343236687]
  An ideal model for dense video captioning should be able to handle long input videos, predict rich, detailed textual descriptions. Current state-of-the-art models process a fixed number of downsampled frames, and make a single full prediction after seeing the whole video. We propose a streaming dense video captioning model that consists of two novel components.
  arXiv  Detail & Related papers  (2024-04-01T17:59:15Z)
• Accurate and Fast Compressed Video Captioning [28.19362369787383]
  Existing video captioning approaches typically require to first sample video frames from a decoded video and then conduct a subsequent process. We study video captioning from a different perspective in compressed domain, which brings multi-fold advantages over the existing pipeline. We propose a simple yet effective end-to-end transformer in the compressed domain for video captioning that enables learning from the compressed video for captioning.
  arXiv  Detail & Related papers  (2023-09-22T13:43:22Z)
• Predictive Coding For Animation-Based Video Compression [13.161311799049978]
  We propose a predictive coding scheme which uses image animation as a predictor, and codes the residual with respect to the actual target frame. Our experiments indicate a significant gain, in excess of 70% compared to the HEVC video standard and over 30% compared to VVC.
  arXiv  Detail & Related papers  (2023-07-09T14:40:54Z)
• Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation [93.18163456287164]
  This paper proposes a novel text-guided video-to-video translation framework to adapt image models to videos. Our framework achieves global style and local texture temporal consistency at a low cost.
  arXiv  Detail & Related papers  (2023-06-13T17:52:23Z)
• Advancing Learned Video Compression with In-loop Frame Prediction [177.67218448278143]
  In this paper, we propose an Advanced Learned Video Compression (ALVC) approach with the in-loop frame prediction module. The predicted frame can serve as a better reference than the previously compressed frame, and therefore it benefits the compression performance. The experiments show the state-of-the-art performance of our ALVC approach in learned video compression.
  arXiv  Detail & Related papers  (2022-11-13T19:53:14Z)
• Recurrent Video Restoration Transformer with Guided Deformable Attention [116.1684355529431]
  We propose RVRT, which processes local neighboring frames in parallel within a globally recurrent framework. RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
  arXiv  Detail & Related papers  (2022-06-05T10:36:09Z)
• VRT: A Video Restoration Transformer [126.79589717404863]
  Video restoration (e.g., video super-resolution) aims to restore high-quality frames from low-quality frames. We propose a Video Restoration Transformer (VRT) with parallel frame prediction and long-range temporal dependency modelling abilities.
  arXiv  Detail & Related papers  (2022-01-28T17:54:43Z)
• SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning [40.556222166309524]
  We present SwinBERT, an end-to-end transformer-based model for video captioning. Our method adopts a video transformer to encode spatial-temporal representations that can adapt to variable lengths of video input. Based on this model architecture, we show that video captioning can benefit significantly from more densely sampled video frames.
  arXiv  Detail & Related papers  (2021-11-25T18:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.