Detail Preserving Residual Feature Pyramid Modules for Optical Flow

• URL: http://arxiv.org/abs/2107.10990v1
• Date: Fri, 23 Jul 2021 01:53:04 GMT
• Title: Detail Preserving Residual Feature Pyramid Modules for Optical Flow
• Authors: Libo Long, Jochen Lang
• Abstract summary: Residual Feature Pyramid Module (RFPM) retains important details in the feature map without changing the overall iterative refinement design of the optical flow estimation. Results show that our RFPM visibly reduces flow errors and improves state-of-art performance in the clean pass of Sintel.
• Score: 3.3301533805099357
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Feature pyramids and iterative refinement have recently led to great progress in optical flow estimation. However, downsampling in feature pyramids can cause blending of foreground objects with the background, which will mislead subsequent decisions in the iterative processing. The results are missing details especially in the flow of thin and of small structures. We propose a novel Residual Feature Pyramid Module (RFPM) which retains important details in the feature map without changing the overall iterative refinement design of the optical flow estimation. RFPM incorporates a residual structure between multiple feature pyramids into a downsampling module that corrects the blending of objects across boundaries. We demonstrate how to integrate our module with two state-of-the-art iterative refinement architectures. Results show that our RFPM visibly reduces flow errors and improves state-of-art performance in the clean pass of Sintel, and is one of the top-performing methods in KITTI. According to the particular modular structure of RFPM, we introduce a special transfer learning approach that can dramatically decrease the training time compared to a typical full optical flow training schedule on multiple datasets.

Related papers

• Motion-Aware Video Frame Interpolation [49.49668436390514]
  We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames. It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
  arXiv  Detail & Related papers  (2024-02-05T11:00:14Z)
• Enhancing Low-light Light Field Images with A Deep Compensation Unfolding Network [52.77569396659629]
  This paper presents the deep compensation network unfolding (DCUNet) for restoring light field (LF) images captured under low-light conditions. The framework uses the intermediate enhanced result to estimate the illumination map, which is then employed in the unfolding process to produce a new enhanced result. To properly leverage the unique characteristics of LF images, this paper proposes a pseudo-explicit feature interaction module.
  arXiv  Detail & Related papers  (2023-08-10T07:53:06Z)
• Structure Flow-Guided Network for Real Depth Super-Resolution [28.63334760296165]
  We propose a novel structure flow-guided depth super-resolution (DSR) framework. A cross-modality flow map is learned to guide the RGB-structure information transferring for precise depth upsampling. Our framework achieves excellent performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-01-31T05:13:55Z)
• GMFlow: Learning Optical Flow via Global Matching [124.57850500778277]
  We propose a GMFlow framework for learning optical flow estimation. It consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. Our new framework outperforms 32-iteration RAFT's performance on the challenging Sintel benchmark.
  arXiv  Detail & Related papers  (2021-11-26T18:59:56Z)
• Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications [78.63280020581662]
  A novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-03-24T08:16:32Z)
• PRAFlow_RVC: Pyramid Recurrent All-Pairs Field Transforms for Optical Flow Estimation in Robust Vision Challenge 2020 [28.77846425802558]
  We present PRAFlow (Pyramid Recurrent All-Pairs Flow), which builds upon the pyramid network structure. Our model was trained on several simulate and real-image datasets, submitted to multiple leaderboards using the same model and parameters, and won the 2nd place in the optical flow task of ECCV 2020 workshop: Robust Vision Challenge.
  arXiv  Detail & Related papers  (2020-09-14T12:27:52Z)
• FDFlowNet: Fast Optical Flow Estimation using a Deep Lightweight Network [12.249680550252327]
  We present a lightweight yet effective model for real-time optical flow estimation, termed FDFlowNet (fast deep flownet) We achieve better or similar accuracy on the challenging KITTI and Sintel benchmarks while being about 2 times faster than PWC-Net.
  arXiv  Detail & Related papers  (2020-06-22T14:01:01Z)
• STC-Flow: Spatio-temporal Context-aware Optical Flow Estimation [13.649566468445258]
  We propose a-temporal contextual network, STC-Flow, for optical flow estimation. STC-Flow contains three key context modules - pyramid spatial context module, temporal context correlation module and residual contextual upsampling module. Experimental results indicate that the proposed scheme achieves the state-of-the-art performance of two-frame based methods on the Sintel dataset and the KITTI 2012/2015 datasets.
  arXiv  Detail & Related papers  (2020-03-01T08:18:57Z)
• FPCR-Net: Feature Pyramidal Correlation and Residual Reconstruction for Optical Flow Estimation [72.41370576242116]
  We propose a semi-supervised Feature Pyramidal Correlation and Residual Reconstruction Network (FPCR-Net) for optical flow estimation from frame pairs. It consists of two main modules: pyramid correlation mapping and residual reconstruction. Experiment results show that the proposed scheme achieves the state-of-the-art performance, with improvement by 0.80, 1.15 and 0.10 in terms of average end-point error (AEE) against competing baseline methods.
  arXiv  Detail & Related papers  (2020-01-17T07:13:51Z)
• A Near-Optimal Gradient Flow for Learning Neural Energy-Based Models [93.24030378630175]
  We propose a novel numerical scheme to optimize the gradient flows for learning energy-based models (EBMs) We derive a second-order Wasserstein gradient flow of the global relative entropy from Fokker-Planck equation. Compared with existing schemes, Wasserstein gradient flow is a smoother and near-optimal numerical scheme to approximate real data densities.
  arXiv  Detail & Related papers  (2019-10-31T02:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.