Error-Aware Spatial Ensembles for Video Frame Interpolation

• URL: http://arxiv.org/abs/2207.12305v1
• Date: Mon, 25 Jul 2022 16:15:38 GMT
• Title: Error-Aware Spatial Ensembles for Video Frame Interpolation
• Authors: Zhixiang Chi, Rasoul Mohammadi Nasiri, Zheng Liu, Yuanhao Yu, Juwei Lu, Jin Tang, Konstantinos N Plataniotis
• Abstract summary: Video frame(VFI) algorithms have improved considerably in recent years due to unprecedented progress in both data-driven algorithms and their implementations. Recent research has introduced advanced motion estimation or novel warping methods as the means to address challenging VFI scenarios. This work introduces such a solution. By closely examining the correlation between optical flow and IE, the paper proposes novel error prediction metrics that partition the middle frame into distinct regions corresponding to different IE levels.
• Score: 50.63021118973639
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video frame interpolation~(VFI) algorithms have improved considerably in recent years due to unprecedented progress in both data-driven algorithms and their implementations. Recent research has introduced advanced motion estimation or novel warping methods as the means to address challenging VFI scenarios. However, none of the published VFI works considers the spatially non-uniform characteristics of the interpolation error (IE). This work introduces such a solution. By closely examining the correlation between optical flow and IE, the paper proposes novel error prediction metrics that partition the middle frame into distinct regions corresponding to different IE levels. Building upon this IE-driven segmentation, and through the use of novel error-controlled loss functions, it introduces an ensemble of spatially adaptive interpolation units that progressively processes and integrates the segmented regions. This spatial ensemble results in an effective and computationally attractive VFI solution. Extensive experimentation on popular video interpolation benchmarks indicates that the proposed solution outperforms the current state-of-the-art (SOTA) in applications of current interest.

Related papers

• Video Frame Interpolation with Region-Distinguishable Priors from SAM [19.350313166180747]
  Region-Distinguishable Priors (RDPs) are represented as spatial-varying Gaussian mixtures. Hierarchical Region-aware Feature Fusion Module (HRFFM) incorporates into various hierarchical stages of VFI's encoder. experiments demonstrate that HRFFM consistently enhances VFI performance across various scenes.
  arXiv  Detail & Related papers  (2023-12-26T03:27:30Z)
• Long-Term Invariant Local Features via Implicit Cross-Domain Correspondences [79.21515035128832]
  We conduct a thorough analysis of the performance of current state-of-the-art feature extraction networks under various domain changes. We propose a novel data-centric method, Implicit Cross-Domain Correspondences (iCDC) iCDC represents the same environment with multiple Neural Radiance Fields, each fitting the scene under individual visual domains.
  arXiv  Detail & Related papers  (2023-11-06T18:53:01Z)
• Surface EMG-Based Inter-Session/Inter-Subject Gesture Recognition by Leveraging Lightweight All-ConvNet and Transfer Learning [17.535392299244066]
  Gesture recognition using low-resolution instantaneous HD-sEMG images opens up new avenues for the development of more fluid and natural muscle-computer interfaces. The data variability between inter-session and inter-subject scenarios presents a great challenge. Existing approaches employed very large and complex deep ConvNet or 2SRNN-based domain adaptation methods to approximate the distribution shift caused by these inter-session and inter-subject data variability. We propose a lightweight All-ConvNet+TL model that leverages lightweight All-ConvNet and transfer learning (TL) for the enhancement of inter-session and inter-subject gesture recognition
  arXiv  Detail & Related papers  (2023-05-13T21:47:55Z)
• Non-Uniform Interpolation in Integrated Gradients for Low-Latency Explainable-AI [2.048335092363435]
  Integrated Gradients (IG) is a popular XAI algorithm that attributes relevance scores to input features. There is a significant computational overhead to generate the explanation which hinders real-time XAI. We propose a novel non-uniform scheme to compute the IG attribution scores which replaces the baseline uniform optimization.
  arXiv  Detail & Related papers  (2023-02-22T03:03:28Z)
• Aligning Correlation Information for Domain Adaptation in Action Recognition [14.586677030468339]
  We propose a novel Adversa Correlation Adaptation Network (ACAN) to align action videos by aligning pixel correlations. ACAN aims to minimize the distribution of correlation information as Pixel Correlation Discrepancy (PCD)
  arXiv  Detail & Related papers  (2021-07-11T00:13:36Z)
• Unsupervised Domain Adaptation for Spatio-Temporal Action Localization [69.12982544509427]
  S-temporal action localization is an important problem in computer vision. We propose an end-to-end unsupervised domain adaptation algorithm. We show that significant performance gain can be achieved when spatial and temporal features are adapted separately or jointly.
  arXiv  Detail & Related papers  (2020-10-19T04:25:10Z)
• Cross-Domain Facial Expression Recognition: A Unified Evaluation Benchmark and Adversarial Graph Learning [85.6386289476598]
  We develop a novel adversarial graph representation adaptation (AGRA) framework for cross-domain holistic-local feature co-adaptation. We conduct extensive and fair evaluations on several popular benchmarks and show that the proposed AGRA framework outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-08-03T15:00:31Z)
• Cross-domain Object Detection through Coarse-to-Fine Feature Adaptation [62.29076080124199]
  This paper proposes a novel coarse-to-fine feature adaptation approach to cross-domain object detection. At the coarse-grained stage, foreground regions are extracted by adopting the attention mechanism, and aligned according to their marginal distributions. At the fine-grained stage, we conduct conditional distribution alignment of foregrounds by minimizing the distance of global prototypes with the same category but from different domains.
  arXiv  Detail & Related papers  (2020-03-23T13:40:06Z)
• Domain Adaptation: Learning Bounds and Algorithms [80.85426994513541]
  We introduce a novel distance between distributions, discrepancy distance, that is tailored to adaptation problems with arbitrary loss functions. We derive novel generalization bounds for domain adaptation for a wide family of loss functions. We also present a series of novel adaptation bounds for large classes of regularization-based algorithms.
  arXiv  Detail & Related papers  (2009-02-19T18:42:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.