Perceptual Coding for Compressed Video Understanding: A New Framework
and Benchmark
- URL: http://arxiv.org/abs/2202.02813v1
- Date: Sun, 6 Feb 2022 16:29:15 GMT
- Title: Perceptual Coding for Compressed Video Understanding: A New Framework
and Benchmark
- Authors: Yuan Tian, Guo Lu, Yichao Yan, Guangtao Zhai, Li Chen, Zhiyong Gao
- Abstract summary: We propose the first coding framework for compressed video understanding, where another learnable perceptual bitstream is introduced and simultaneously transported with the video bitstream.
Our framework can enjoy the best of both two worlds, (1) highly efficient content-coding of industrial video and (2) flexible perceptual-coding of neural networks (NNs)
- Score: 57.23523738351178
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Most video understanding methods are learned on high-quality videos. However,
in most real-world scenarios, the videos are first compressed before the
transportation and then decompressed for understanding. The decompressed videos
are degraded in terms of perceptual quality, which may degenerate the
downstream tasks. To address this issue, we propose the first coding framework
for compressed video understanding, where another learnable perceptual
bitstream is introduced and simultaneously transported with the video
bitstream. With the sophisticatedly designed optimization target and network
architectures, this new stream largely boosts the perceptual quality of the
decoded videos yet with a small bit cost. Our framework can enjoy the best of
both two worlds, (1) highly efficient content-coding of industrial video codec
and (2) flexible perceptual-coding of neural networks (NNs). Finally, we build
a rigorous benchmark for compressed video understanding over four different
compression levels, six large-scale datasets, and two popular tasks. The
proposed Dual-bitstream Perceptual Video Coding framework Dual-PVC consistently
demonstrates significantly stronger performances than the baseline codec under
the same bitrate level.
Related papers
- Standard compliant video coding using low complexity, switchable neural wrappers [8.149130379436759]
We propose a new framework featuring standard compatibility, high performance, and low decoding complexity.
We employ a set of jointly optimized neural pre- and post-processors, wrapping a standard video, to encode videos at different resolutions.
We design a low complexity neural post-processor architecture that can handle different upsampling ratios.
arXiv Detail & Related papers (2024-07-10T06:36:45Z) - Lightweight Hybrid Video Compression Framework Using Reference-Guided
Restoration Network [12.033330902093152]
We propose a new lightweight hybrid video consisting of a conventional video(HEVC / VVC), a lossless image, and our new restoration network.
Our method achieves comparable performance to top-tier methods, even when applied to HEVC.
arXiv Detail & Related papers (2023-03-21T04:42:44Z) - Sandwiched Video Compression: Efficiently Extending the Reach of
Standard Codecs with Neural Wrappers [11.968545394054816]
We propose a video compression system that wraps neural networks around a standard video.
Networks are trained jointly to optimize a rate-distortion loss function.
We observe 30% improvements in rate at the same quality over HEVC.
arXiv Detail & Related papers (2023-03-20T22:03:44Z) - Compressed Vision for Efficient Video Understanding [83.97689018324732]
We propose a framework enabling research on hour-long videos with the same hardware that can now process second-long videos.
We replace standard video compression, e.g. JPEG, with neural compression and show that we can directly feed compressed videos as inputs to regular video networks.
arXiv Detail & Related papers (2022-10-06T15:35:49Z) - Leveraging Bitstream Metadata for Fast, Accurate, Generalized Compressed
Video Quality Enhancement [74.1052624663082]
We develop a deep learning architecture capable of restoring detail to compressed videos.
We show that this improves restoration accuracy compared to prior compression correction methods.
We condition our model on quantization data which is readily available in the bitstream.
arXiv Detail & Related papers (2022-01-31T18:56:04Z) - Variable Rate Video Compression using a Hybrid Recurrent Convolutional
Learning Framework [1.9290392443571382]
This paper presents PredEncoder, a hybrid video compression framework based on the concept of predictive auto-encoding.
A variable-rate block encoding scheme has been proposed in the paper that leads to remarkably high quality to bit-rate ratios.
arXiv Detail & Related papers (2020-04-08T20:49:25Z) - Content Adaptive and Error Propagation Aware Deep Video Compression [110.31693187153084]
We propose a content adaptive and error propagation aware video compression system.
Our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame.
Instead of using the hand-crafted coding modes in the traditional compression systems, we design an online encoder updating scheme in our system.
arXiv Detail & Related papers (2020-03-25T09:04:24Z) - Learning for Video Compression with Hierarchical Quality and Recurrent
Enhancement [164.7489982837475]
We propose a Hierarchical Learned Video Compression (HLVC) method with three hierarchical quality layers and a recurrent enhancement network.
In our HLVC approach, the hierarchical quality benefits the coding efficiency, since the high quality information facilitates the compression and enhancement of low quality frames at encoder and decoder sides.
arXiv Detail & Related papers (2020-03-04T09:31:37Z) - Video Coding for Machines: A Paradigm of Collaborative Compression and
Intelligent Analytics [127.65410486227007]
Video coding, which targets to compress and reconstruct the whole frame, and feature compression, which only preserves and transmits the most critical information, stand at two ends of the scale.
Recent endeavors in imminent trends of video compression, e.g. deep learning based coding tools and end-to-end image/video coding, and MPEG-7 compact feature descriptor standards, promote the sustainable and fast development in their own directions.
In this paper, thanks to booming AI technology, e.g. prediction and generation models, we carry out exploration in the new area, Video Coding for Machines (VCM), arising from the emerging MPEG
arXiv Detail & Related papers (2020-01-10T17:24:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.