Online Video Quality Enhancement with Spatial-Temporal Look-up Tables

• URL: http://arxiv.org/abs/2311.13616v2
• Date: Wed, 10 Jul 2024 14:06:16 GMT
• Title: Online Video Quality Enhancement with Spatial-Temporal Look-up Tables
• Authors: Zefan Qu, Xinyang Jiang, Yifan Yang, Dongsheng Li, Cairong Zhao,
• Abstract summary: Low latency rates are crucial for online video-based applications, such as video conferencing and cloud gaming. Existing quality enhancement methods are limited by slow inference speed and the requirement for temporal information contained in future frames. We propose STLVQE, specifically designed to address the rarely studied online video quality enhancement (Online-VQE) problem.
• Score: 42.07242907586958
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low latency rates are crucial for online video-based applications, such as video conferencing and cloud gaming, which make improving video quality in online scenarios increasingly important. However, existing quality enhancement methods are limited by slow inference speed and the requirement for temporal information contained in future frames, making it challenging to deploy them directly in online tasks. In this paper, we propose a novel method, STLVQE, specifically designed to address the rarely studied online video quality enhancement (Online-VQE) problem. Our STLVQE designs a new VQE framework which contains a Module-Agnostic Feature Extractor that greatly reduces the redundant computations and redesign the propagation, alignment, and enhancement module of the network. A Spatial-Temporal Look-up Tables (STL) is proposed, which extracts spatial-temporal information in videos while saving substantial inference time. To the best of our knowledge, we are the first to exploit the LUT structure to extract temporal information in video tasks. Extensive experiments on the MFQE 2.0 dataset demonstrate that our STLVQE achieves a satisfactory performance-speed trade-off.

Related papers

• LMM-VQA: Advancing Video Quality Assessment with Large Multimodal Models [53.64461404882853]
  Video quality assessment (VQA) algorithms are needed to monitor and optimize the quality of streaming videos. Here, we propose the first Large Multi-Modal Video Quality Assessment (LMM-VQA) model, which introduces a novel visual modeling strategy for quality-aware feature extraction.
  arXiv  Detail & Related papers  (2024-08-26T04:29:52Z)
• Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision [24.568643475808564]
  Video Self-Training with augmented Reasoning (Video-STaR) is the first video self-training approach. Video-STaR allows the utilization of any labeled video dataset for video instruction tuning.
  arXiv  Detail & Related papers  (2024-07-08T17:59:42Z)
• STIP: A SpatioTemporal Information-Preserving and Perception-Augmented Model for High-Resolution Video Prediction [78.129039340528]
  We propose a Stemporal Information-Preserving and Perception-Augmented Model (STIP) to solve the above two problems. The proposed model aims to preserve thetemporal information for videos during the feature extraction and the state transitions. Experimental results show that the proposed STIP can predict videos with more satisfactory visual quality compared with a variety of state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-06-09T09:49:04Z)
• Fast Online Video Super-Resolution with Deformable Attention Pyramid [172.16491820970646]
  Video super-resolution (VSR) has many applications that pose strict causal, real-time, and latency constraints, including video streaming and TV. We propose a recurrent VSR architecture based on a deformable attention pyramid (DAP)
  arXiv  Detail & Related papers  (2022-02-03T17:49:04Z)
• Enhanced Spatio-Temporal Interaction Learning for Video Deraining: A Faster and Better Framework [93.37833982180538]
  Video deraining is an important task in computer vision as the unwanted rain hampers the visibility of videos and deteriorates the robustness of most outdoor vision systems. We present a new end-to-end deraining framework, named Enhanced Spatio-Temporal Interaction Network (ESTINet) ESTINet considerably boosts current state-of-the-art video deraining quality and speed.
  arXiv  Detail & Related papers  (2021-03-23T05:19:35Z)
• RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content [44.03188436272383]
  We introduce an effective and efficient video quality model for content, which we dub the Rapid and Accurate Video Quality Evaluator (RAPIQUE) RAPIQUE combines and leverages the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features. Our experimental results on recent large-scale video quality databases show that RAPIQUE delivers top performances on all the datasets at a considerably lower computational expense.
  arXiv  Detail & Related papers  (2021-01-26T17:23:46Z)
• An Efficient Recurrent Adversarial Framework for Unsupervised Real-Time Video Enhancement [132.60976158877608]
  We propose an efficient adversarial video enhancement framework that learns directly from unpaired video examples. In particular, our framework introduces new recurrent cells that consist of interleaved local and global modules for implicit integration of spatial and temporal information. The proposed design allows our recurrent cells to efficiently propagate-temporal-information across frames and reduces the need for high complexity networks.
  arXiv  Detail & Related papers  (2020-12-24T00:03:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.