Learned Scalable Video Coding For Humans and Machines
- URL: http://arxiv.org/abs/2307.08978v1
- Date: Tue, 18 Jul 2023 05:22:25 GMT
- Title: Learned Scalable Video Coding For Humans and Machines
- Authors: Hadi Hadizadeh and Ivan V. Baji\'c
- Abstract summary: We introduce the first end-to-end learnable video that supports a machine vision task in its base layer, while its enhancement layer supports input reconstruction for human viewing.
Our framework outperforms both state-of-the-art learned and conventional video codecs in its base layer, while maintaining comparable performance on the human vision task in its enhancement layer.
- Score: 39.32955669909719
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Video coding has traditionally been developed to support services such as
video streaming, videoconferencing, digital TV, and so on. The main intent was
to enable human viewing of the encoded content. However, with the advances in
deep neural networks (DNNs), encoded video is increasingly being used for
automatic video analytics performed by machines. In applications such as
automatic traffic monitoring, analytics such as vehicle detection, tracking and
counting, would run continuously, while human viewing could be required
occasionally to review potential incidents. To support such applications, a new
paradigm for video coding is needed that will facilitate efficient
representation and compression of video for both machine and human use in a
scalable manner. In this manuscript, we introduce the first end-to-end
learnable video codec that supports a machine vision task in its base layer,
while its enhancement layer supports input reconstruction for human viewing.
The proposed system is constructed based on the concept of conditional coding
to achieve better compression gains. Comprehensive experimental evaluations
conducted on four standard video datasets demonstrate that our framework
outperforms both state-of-the-art learned and conventional video codecs in its
base layer, while maintaining comparable performance on the human vision task
in its enhancement layer. We will provide the implementation of the proposed
system at www.github.com upon completion of the review process.
Related papers
- NN-VVC: Versatile Video Coding boosted by self-supervisedly learned
image coding for machines [19.183883119933558]
This paper proposes a hybrid for machines called NN-VVC, which combines the advantages of an E2E-learned image and a CVC to achieve high performance in both image and video coding for machines.
Our experiments show that the proposed system achieved up to -43.20% and -26.8% Bjontegaard Delta rate reduction over VVC for image and video data, respectively.
arXiv Detail & Related papers (2024-01-19T15:33:46Z) - VNVC: A Versatile Neural Video Coding Framework for Efficient
Human-Machine Vision [59.632286735304156]
It is more efficient to enhance/analyze the coded representations directly without decoding them into pixels.
We propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis.
arXiv Detail & Related papers (2023-06-19T03:04:57Z) - Task Oriented Video Coding: A Survey [0.5076419064097732]
State-of-the-art video coding standards, such as H.265/HEVC and Versatile Video Coding, are still designed with the assumption the compressed video will be watched by humans.
With the tremendous advance and maturation of deep neural networks in solving computer vision tasks, more and more videos are directly analyzed by deep neural networks without humans' involvement.
We explore and summarize recent progress on computer vision task oriented video coding and emerging video coding standard, Video Coding for Machines.
arXiv Detail & Related papers (2022-08-15T16:21:54Z) - Scalable Video Coding for Humans and Machines [42.870358996305356]
We propose a scalable video coding framework that supports machine vision through its base layer bitstream and human vision via its enhancement layer bitstream.
The proposed framework includes components from both conventional and Deep Neural Network (DNN)-based video coding.
arXiv Detail & Related papers (2022-08-04T07:45:41Z) - Video Exploration via Video-Specific Autoencoders [60.256055890647595]
We present video-specific autoencoders that enables human-controllable video exploration.
We observe that a simple autoencoder trained on multiple frames of a specific video enables one to perform a large variety of video processing and editing tasks.
arXiv Detail & Related papers (2021-03-31T17:56:13Z) - Human-Machine Collaborative Video Coding Through Cuboidal Partitioning [26.70051123157869]
We propose a video coding framework by leveraging on to the commonality that exists between human vision and machine vision applications using cuboids.
Cuboids, estimated rectangular regions over a video frame, are computationally efficient, has a compact representation and object centric.
Herein cuboidal feature descriptors are extracted from the current frame and then employed for accomplishing a machine vision task in the form of object detection.
arXiv Detail & Related papers (2021-02-02T04:44:45Z) - Video Coding for Machines: A Paradigm of Collaborative Compression and
Intelligent Analytics [127.65410486227007]
Video coding, which targets to compress and reconstruct the whole frame, and feature compression, which only preserves and transmits the most critical information, stand at two ends of the scale.
Recent endeavors in imminent trends of video compression, e.g. deep learning based coding tools and end-to-end image/video coding, and MPEG-7 compact feature descriptor standards, promote the sustainable and fast development in their own directions.
In this paper, thanks to booming AI technology, e.g. prediction and generation models, we carry out exploration in the new area, Video Coding for Machines (VCM), arising from the emerging MPEG
arXiv Detail & Related papers (2020-01-10T17:24:13Z) - An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond
Feature and Signal [99.49099501559652]
Video Coding for Machine (VCM) aims to bridge the gap between visual feature compression and classical video coding.
We employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern.
By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames.
arXiv Detail & Related papers (2020-01-09T14:18:18Z) - Towards Coding for Human and Machine Vision: A Scalable Image Coding
Approach [104.02201472370801]
We come up with a novel image coding framework by leveraging both the compressive and the generative models.
By introducing advanced generative models, we train a flexible network to reconstruct images from compact feature representations and the reference pixels.
Experimental results demonstrate the superiority of our framework in both human visual quality and facial landmark detection.
arXiv Detail & Related papers (2020-01-09T10:37:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.