EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training

• URL: http://arxiv.org/abs/2411.16312v1
• Date: Mon, 25 Nov 2024 12:01:57 GMT
• Title: EPS: Efficient Patch Sampling for Video Overfitting in Deep Super-Resolution Model Training
• Authors: Yiying Wei, Hadi Amirpour, Jong Hwan Ko, Christian Timmerer,
• Abstract summary: We propose an efficient patch sampling method named EPS for video SR network overfitting. Our method reduces the number of patches for the training to 4% to 25%, depending on the resolution and number of clusters. Compared to the state-of-the-art patch sampling method, EMT, our approach achieves an 83% decrease in overall run time.
• Score: 15.684865589513597
• License:
• Abstract: Leveraging the overfitting property of deep neural networks (DNNs) is trending in video delivery systems to enhance quality within bandwidth limits. Existing approaches transmit overfitted super-resolution (SR) model streams for low-resolution (LR) bitstreams, which are used to reconstruct high-resolution (HR) videos at the decoder. Although these approaches show promising results, the huge computational costs of training a large number of video frames limit their practical applications. To overcome this challenge, we propose an efficient patch sampling method named EPS for video SR network overfitting, which identifies the most valuable training patches from video frames. To this end, we first present two low-complexity Discrete Cosine Transform (DCT)-based spatial-temporal features to measure the complexity score of each patch directly. By analyzing the histogram distribution of these features, we then categorize all possible patches into different clusters and select training patches from the cluster with the highest spatial-temporal information. The number of sampled patches is adaptive based on the video content, addressing the trade-off between training complexity and efficiency. Our method reduces the number of patches for the training to 4% to 25%, depending on the resolution and number of clusters, while maintaining high video quality and significantly enhancing training efficiency. Compared to the state-of-the-art patch sampling method, EMT, our approach achieves an 83% decrease in overall run time.

Related papers

• Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs [56.040198387038025]
  We present a novel prompt-guided visual perception framework (abbreviated as Free Video-LLM) for efficient inference of training-free video LLMs. Our method effectively reduces the number of visual tokens while maintaining high performance across multiple video question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T12:35:12Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• PatchScaler: An Efficient Patch-Independent Diffusion Model for Image Super-Resolution [44.345740602726345]
  PatchScaler is an efficient patch-independent diffusion pipeline for single image super-resolution. A texture prompt adaptively retrieves texture priors for the target patch from a common reference texture memory. Our code achieves superior performance in both quantitative and qualitative evaluations, while significantly speeding up inference.
  arXiv  Detail & Related papers  (2024-05-27T13:31:46Z)
• Towards High-Quality and Efficient Video Super-Resolution via Spatial-Temporal Data Overfitting [27.302681897961588]
  Deep convolutional neural networks (DNNs) are widely used in various fields of computer vision. We propose a novel method for high-quality and efficient video resolution upscaling tasks. We deploy our models on an off-the-shelf mobile phone, and experimental results show that our method achieves real-time video super-resolution with high video quality.
  arXiv  Detail & Related papers  (2023-03-15T02:40:02Z)
• Neighbourhood Representative Sampling for Efficient End-to-end Video Quality Assessment [60.57703721744873]
  The increased resolution of real-world videos presents a dilemma between efficiency and accuracy for deep Video Quality Assessment (VQA) In this work, we propose a unified scheme, spatial-temporal grid mini-cube sampling (St-GMS) to get a novel type of sample, named fragments. With fragments and FANet, the proposed efficient end-to-end FAST-VQA and FasterVQA achieve significantly better performance than existing approaches on all VQA benchmarks.
  arXiv  Detail & Related papers  (2022-10-11T11:38:07Z)
• Efficient Meta-Tuning for Content-aware Neural Video Delivery [40.3731358963689]
  We present Efficient Meta-Tuning (EMT) to reduce the computational cost. EMT adapts a meta-learned model to the first chunk of the input video. We propose a novel sampling strategy to extract the most challenging patches from video frames.
  arXiv  Detail & Related papers  (2022-07-20T06:47:10Z)
• FAST-VQA: Efficient End-to-end Video Quality Assessment with Fragment Sampling [54.31355080688127]
  Current deep video quality assessment (VQA) methods are usually with high computational costs when evaluating high-resolution videos. We propose Grid Mini-patch Sampling (GMS), which allows consideration of local quality by sampling patches at their raw resolution. We build the Fragment Attention Network (FANet) specially designed to accommodate fragments as inputs. FAST-VQA improves state-of-the-art accuracy by around 10% while reducing 99.5% FLOPs on 1080P high-resolution videos.
  arXiv  Detail & Related papers  (2022-07-06T11:11:43Z)
• Capturing Temporal Information in a Single Frame: Channel Sampling Strategies for Action Recognition [19.220288614585147]
  We address the problem of capturing temporal information for video classification in 2D networks, without increasing computational cost. We propose a novel sampling strategy, where we re-order the channels of the input video, to capture short-term frame-to-frame changes. Our sampling strategies do not require training from scratch and do not increase the computational cost of training and testing.
  arXiv  Detail & Related papers  (2022-01-25T15:24:37Z)
• Investigating Tradeoffs in Real-World Video Super-Resolution [90.81396836308085]
  Real-world video super-resolution (VSR) models are often trained with diverse degradations to improve generalizability. To alleviate the first tradeoff, we propose a degradation scheme that reduces up to 40% of training time without sacrificing performance. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences.
  arXiv  Detail & Related papers  (2021-11-24T18:58:21Z)
• Deep Space-Time Video Upsampling Networks [47.62807427163614]
  Video super-resolution (VSR) and frame (FI) are traditional computer vision problems. We propose an end-to-end framework for the space-time video upsampling by efficiently merging VSR and FI into a joint framework. Results show better results both quantitatively and qualitatively, while reducing the time (x7 faster) and the number of parameters (30%) compared to baselines.
  arXiv  Detail & Related papers  (2020-04-06T07:04:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.