FPCR-Net: Feature Pyramidal Correlation and Residual Reconstruction for Optical Flow Estimation

• URL: http://arxiv.org/abs/2001.06171v4
• Date: Tue, 23 Nov 2021 04:08:25 GMT
• Title: FPCR-Net: Feature Pyramidal Correlation and Residual Reconstruction for Optical Flow Estimation
• Authors: Xiaolin Song, Yuyang Zhao, Jingyu Yang, Cuiling Lan, and Wenjun Zeng
• Abstract summary: We propose a semi-supervised Feature Pyramidal Correlation and Residual Reconstruction Network (FPCR-Net) for optical flow estimation from frame pairs. It consists of two main modules: pyramid correlation mapping and residual reconstruction. Experiment results show that the proposed scheme achieves the state-of-the-art performance, with improvement by 0.80, 1.15 and 0.10 in terms of average end-point error (AEE) against competing baseline methods.
• Score: 72.41370576242116
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optical flow estimation is an important yet challenging problem in the field of video analytics. The features of different semantics levels/layers of a convolutional neural network can provide information of different granularity. To exploit such flexible and comprehensive information, we propose a semi-supervised Feature Pyramidal Correlation and Residual Reconstruction Network (FPCR-Net) for optical flow estimation from frame pairs. It consists of two main modules: pyramid correlation mapping and residual reconstruction. The pyramid correlation mapping module takes advantage of the multi-scale correlations of global/local patches by aggregating features of different scales to form a multi-level cost volume. The residual reconstruction module aims to reconstruct the sub-band high-frequency residuals of finer optical flow in each stage. Based on the pyramid correlation mapping, we further propose a correlation-warping-normalization (CWN) module to efficiently exploit the correlation dependency. Experiment results show that the proposed scheme achieves the state-of-the-art performance, with improvement by 0.80, 1.15 and 0.10 in terms of average end-point error (AEE) against competing baseline methods - FlowNet2, LiteFlowNet and PWC-Net on the Final pass of Sintel dataset, respectively.

Related papers

• DA-Flow: Dual Attention Normalizing Flow for Skeleton-based Video Anomaly Detection [52.74152717667157]
  We propose a lightweight module called Dual Attention Module (DAM) for capturing cross-dimension interaction relationships in-temporal skeletal data. It employs the frame attention mechanism to identify the most significant frames and the skeleton attention mechanism to capture broader relationships across fixed partitions with minimal parameters and flops.
  arXiv  Detail & Related papers  (2024-06-05T06:18:03Z)
• Flow Guidance Deformable Compensation Network for Video Frame Interpolation [33.106776459443275]
  We propose a flow guidance deformable compensation network (FGDCN) to overcome the drawbacks of existing motion-based methods. FGDCN decomposes the frame sampling process into two steps: a flow step and a deformation step. Experimental results show that the proposed algorithm achieves excellent performance on various datasets with fewer parameters.
  arXiv  Detail & Related papers  (2022-11-22T09:35:14Z)
• DeMFI: Deep Joint Deblurring and Multi-Frame Interpolation with Flow-Guided Attentive Correlation and Recursive Boosting [50.17500790309477]
  DeMFI-Net is a joint deblurring and multi-frame framework. It converts blurry videos of lower-frame-rate to sharp videos at higher-frame-rate. It achieves state-of-the-art (SOTA) performances for diverse datasets.
  arXiv  Detail & Related papers  (2021-11-19T00:00:15Z)
• Enhanced Correlation Matching based Video Frame Interpolation [5.304928339627251]
  We propose a novel framework called the Enhanced Correlation Matching based Video Frame Interpolation Network. The proposed scheme employs the recurrent pyramid architecture that shares the parameters among each pyramid layer for optical flow estimation. Experiment results demonstrate that the proposed scheme outperforms the previous works at 4K video data and low-resolution benchmark datasets as well as in terms of objective and subjective quality.
  arXiv  Detail & Related papers  (2021-11-17T02:43:45Z)
• Detail Preserving Residual Feature Pyramid Modules for Optical Flow [3.3301533805099357]
  Residual Feature Pyramid Module (RFPM) retains important details in the feature map without changing the overall iterative refinement design of the optical flow estimation. Results show that our RFPM visibly reduces flow errors and improves state-of-art performance in the clean pass of Sintel.
  arXiv  Detail & Related papers  (2021-07-23T01:53:04Z)
• A^2-FPN: Attention Aggregation based Feature Pyramid Network for Instance Segmentation [68.10621089649486]
  We propose Attention Aggregation based Feature Pyramid Network (A2-FPN) to improve multi-scale feature learning. A2-FPN achieves an improvement of 2.0% and 1.4% mask AP when integrated into the strong baselines such as Cascade Mask R-CNN and Hybrid Task Cascade.
  arXiv  Detail & Related papers  (2021-05-07T11:51:08Z)
• PRAFlow_RVC: Pyramid Recurrent All-Pairs Field Transforms for Optical Flow Estimation in Robust Vision Challenge 2020 [28.77846425802558]
  We present PRAFlow (Pyramid Recurrent All-Pairs Flow), which builds upon the pyramid network structure. Our model was trained on several simulate and real-image datasets, submitted to multiple leaderboards using the same model and parameters, and won the 2nd place in the optical flow task of ECCV 2020 workshop: Robust Vision Challenge.
  arXiv  Detail & Related papers  (2020-09-14T12:27:52Z)
• Out-of-distribution Generalization via Partial Feature Decorrelation [72.96261704851683]
  We present a novel Partial Feature Decorrelation Learning (PFDL) algorithm, which jointly optimize a feature decomposition network and the target image classification model. The experiments on real-world datasets demonstrate that our method can improve the backbone model's accuracy on OOD image classification datasets.
  arXiv  Detail & Related papers  (2020-07-30T05:48:48Z)
• Augmented Parallel-Pyramid Net for Attention Guided Pose-Estimation [90.28365183660438]
  This paper proposes an augmented parallel-pyramid net with attention partial module and differentiable auto-data augmentation. We define a new pose search space where the sequences of data augmentations are formulated as a trainable and operational CNN component. Notably, our method achieves the top-1 accuracy on the challenging COCO keypoint benchmark and the state-of-the-art results on the MPII datasets.
  arXiv  Detail & Related papers  (2020-03-17T03:52:17Z)
• STC-Flow: Spatio-temporal Context-aware Optical Flow Estimation [13.649566468445258]
  We propose a-temporal contextual network, STC-Flow, for optical flow estimation. STC-Flow contains three key context modules - pyramid spatial context module, temporal context correlation module and residual contextual upsampling module. Experimental results indicate that the proposed scheme achieves the state-of-the-art performance of two-frame based methods on the Sintel dataset and the KITTI 2012/2015 datasets.
  arXiv  Detail & Related papers  (2020-03-01T08:18:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.