Task-Aware Encoder Control for Deep Video Compression

• URL: http://arxiv.org/abs/2404.04848v2
• Date: Sat, 20 Apr 2024 07:54:18 GMT
• Title: Task-Aware Encoder Control for Deep Video Compression
• Authors: Xingtong Ge, Jixiang Luo, Xinjie Zhang, Tongda Xu, Guo Lu, Dailan He, Jing Geng, Yan Wang, Jun Zhang, Hongwei Qin,
• Abstract summary: We introduce an innovative encoder controller for deep video compression for machines. This controller features a mode prediction and a Group of Pictures (GoP) selection module. Our approach centralizes control at the encoding stage, allowing adjustments across different tasks.
• Score: 26.778793247958053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior research on deep video compression (DVC) for machine tasks typically necessitates training a unique codec for each specific task, mandating a dedicated decoder per task. In contrast, traditional video codecs employ a flexible encoder controller, enabling the adaptation of a single codec to different tasks through mechanisms like mode prediction. Drawing inspiration from this, we introduce an innovative encoder controller for deep video compression for machines. This controller features a mode prediction and a Group of Pictures (GoP) selection module. Our approach centralizes control at the encoding stage, allowing for adaptable encoder adjustments across different tasks, such as detection and tracking, while maintaining compatibility with a standard pre-trained DVC decoder. Empirical evidence demonstrates that our method is applicable across multiple tasks with various existing pre-trained DVCs. Moreover, extensive experiments demonstrate that our method outperforms previous DVC by about 25% bitrate for different tasks, with only one pre-trained decoder.

Related papers

• DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models [22.276574156358084]
  We build a multi-exit encoder-decoder transformer model which is trained with deep supervision so that each of its decoder layers is capable of generating plausible predictions. We show our approach can reduce overall inference latency by 30%-60% with comparable or even higher accuracy compared to baselines.
  arXiv  Detail & Related papers  (2023-11-15T01:01:02Z)
• Deep Video Codec Control for Vision Models [33.95098277668838]
  We show that standard-coded videos vastly deteriorate the performance of deep vision models. This paper presents the first end-to-end learnable deep video control that considers both bandwidth constraints and downstream deep vision performance.
  arXiv  Detail & Related papers  (2023-08-30T16:44:38Z)
• A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision [93.90545426665999]
  We take a close look at autoregressive decoders for multi-task learning in multimodal computer vision. A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well. It can be seen as teaching a decoder to interact with a pretrained vision model via natural language.
  arXiv  Detail & Related papers  (2023-03-30T13:42:58Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Efficient VVC Intra Prediction Based on Deep Feature Fusion and Probability Estimation [57.66773945887832]
  We propose to optimize Versatile Video Coding (VVC) complexity at intra-frame prediction, with a two-stage framework of deep feature fusion and probability estimation. Experimental results on standard database demonstrate the superiority of proposed method, especially for High Definition (HD) and Ultra-HD (UHD) video sequences.
  arXiv  Detail & Related papers  (2022-05-07T08:01:32Z)
• Multitask Learning for VVC Quality Enhancement and Super-Resolution [11.446576112498596]
  We propose a learning-based solution as a post-processing step to enhance the decoded VVC video quality. Our method relies on multitask learning to perform both quality enhancement and super-resolution using a single shared network optimized for multiple levels.
  arXiv  Detail & Related papers  (2021-04-16T19:05:26Z)
• Video Exploration via Video-Specific Autoencoders [60.256055890647595]
  We present video-specific autoencoders that enables human-controllable video exploration. We observe that a simple autoencoder trained on multiple frames of a specific video enables one to perform a large variety of video processing and editing tasks.
  arXiv  Detail & Related papers  (2021-03-31T17:56:13Z)
• Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network [99.03895740754402]
  We propose a two-stream decoupled design of encoder-decoder structure, in which two decoupled cross-modal encoder and decoder are involved. As an alternative, we propose a primary scheduled sampling strategy that mitigates such discrepancy via pretraining encoder-decoder in a two-pass manner.
  arXiv  Detail & Related papers  (2021-01-27T17:36:57Z)
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics [127.65410486227007]
  Video coding, which targets to compress and reconstruct the whole frame, and feature compression, which only preserves and transmits the most critical information, stand at two ends of the scale. Recent endeavors in imminent trends of video compression, e.g. deep learning based coding tools and end-to-end image/video coding, and MPEG-7 compact feature descriptor standards, promote the sustainable and fast development in their own directions. In this paper, thanks to booming AI technology, e.g. prediction and generation models, we carry out exploration in the new area, Video Coding for Machines (VCM), arising from the emerging MPEG
  arXiv  Detail & Related papers  (2020-01-10T17:24:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.