Benchmarking the Robustness of Optical Flow Estimation to Corruptions

• URL: http://arxiv.org/abs/2411.14865v1
• Date: Fri, 22 Nov 2024 11:31:01 GMT
• Title: Benchmarking the Robustness of Optical Flow Estimation to Corruptions
• Authors: Zhonghua Yi, Hao Shi, Qi Jiang, Yao Gao, Ze Wang, Yufan Zhang, Kailun Yang, Kaiwei Wang,
• Abstract summary: We introduce 7 temporal corruptions specifically designed for the benchmarking of optical flow models. Two robustness benchmarks, KITTI-FC and GoPro-FC, are established as the first corruption benchmark for optical flow estimation. 29 model variants from 15 optical flow methods are evaluated, yielding 10 intriguing observations.
• Score: 25.789811424859554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optical flow estimation is extensively used in autonomous driving and video editing. While existing models demonstrate state-of-the-art performance across various benchmarks, the robustness of these methods has been infrequently investigated. Despite some research focusing on the robustness of optical flow models against adversarial attacks, there has been a lack of studies investigating their robustness to common corruptions. Taking into account the unique temporal characteristics of optical flow, we introduce 7 temporal corruptions specifically designed for benchmarking the robustness of optical flow models, in addition to 17 classical single-image corruptions, in which advanced PSF Blur simulation method is performed. Two robustness benchmarks, KITTI-FC and GoPro-FC, are subsequently established as the first corruption robustness benchmark for optical flow estimation, with Out-Of-Domain (OOD) and In-Domain (ID) settings to facilitate comprehensive studies. Robustness metrics, Corruption Robustness Error (CRE), Corruption Robustness Error ratio (CREr), and Relative Corruption Robustness Error (RCRE) are further introduced to quantify the optical flow estimation robustness. 29 model variants from 15 optical flow methods are evaluated, yielding 10 intriguing observations, such as 1) the absolute robustness of the model is heavily dependent on the estimation performance; 2) the corruptions that diminish local information are more serious than that reduce visual effects. We also give suggestions for the design and application of optical flow models. We anticipate that our benchmark will serve as a foundational resource for advancing research in robust optical flow estimation. The benchmarks and source code will be released at https://github.com/ZhonghuaYi/optical_flow_robustness_benchmark.

Related papers

• Examining the Impact of Optical Aberrations to Image Classification and Object Detection Models [58.98742597810023]
  Vision models have to behave in a robust way to disturbances such as noise or blur. This paper studies two datasets of blur corruptions, which we denote OpticsBench and LensCorruptions. Evaluations for image classification and object detection on ImageNet and MSCOCO show that for a variety of different pre-trained models, the performance on OpticsBench and LensCorruptions varies significantly.
  arXiv  Detail & Related papers  (2025-04-25T17:23:47Z)
• DPFlow: Adaptive Optical Flow Estimation with a Dual-Pyramid Framework [57.69159159559054]
  We propose DPFlow, an adaptive optical flow architecture capable of generalizing up to 8K resolution inputs. We also introduce Kubric-NK, a new benchmark for evaluating optical flow methods with input resolutions ranging from 1K to 8K.
  arXiv  Detail & Related papers  (2025-03-19T04:18:18Z)
• Efficient Diffusion as Low Light Enhancer [63.789138528062225]
  Reflectance-Aware Trajectory Refinement (RATR) is a simple yet effective module to refine the teacher trajectory using the reflectance component of images. textbfReflectance-aware textbfDiffusion with textbfDistilled textbfTrajectory (textbfReDDiT) is an efficient and flexible distillation framework tailored for Low-Light Image Enhancement (LLIE)
  arXiv  Detail & Related papers  (2024-10-16T08:07:18Z)
• SciFlow: Empowering Lightweight Optical Flow Models with Self-Cleaning Iterations [44.92134227376008]
  This paper introduces two synergistic techniques, Self-Cleaning Iteration (SCI) and Regression Focal Loss (RFL) SCI and RFL prove particularly effective in mitigating error propagation, a prevalent issue in optical flow models that employ iterative refinement. The effectiveness of our proposed SCI and RFL techniques, collectively referred to as SciFlow for brevity, is demonstrated across two distinct lightweight optical flow model architectures in our experiments.
  arXiv  Detail & Related papers  (2024-04-11T21:41:55Z)
• Benchmarking the Robustness of Temporal Action Detection Models Against Temporal Corruptions [21.031546310537692]
  Temporal action detection (TAD) aims to locate action positions and recognize action categories in long-term untrimmed videos. We observe that temporal information in videos can be occasionally corrupted, such as missing or blurred frames. In this paper, we extensively analyze the robustness of seven leading TAD methods and obtain some interesting findings.
  arXiv  Detail & Related papers  (2024-03-29T16:01:00Z)
• OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation [55.676358801492114]
  We propose OCAI, a method that supports robust frame ambiguities by generating intermediate video frames alongside optical flows in between. Our evaluations demonstrate superior quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.
  arXiv  Detail & Related papers  (2024-03-26T20:23:48Z)
• DepthFM: Fast Monocular Depth Estimation with Flow Matching [22.206355073676082]
  Current discriminative approaches to this problem are limited due to blurry artifacts. State-of-the-art generative methods suffer from slow sampling due to their SDE nature. We observe that this can be effectively framed using flow matching, since its straight trajectories through solution space offer efficiency and high quality.
  arXiv  Detail & Related papers  (2024-03-20T17:51:53Z)
• A Perturbation Constrained Adversarial Attack for Evaluating the Robustness of Optical Flow [0.0]
  Perturbation Constrained Flow Attack (PCFA) is a novel adversarial attack that emphasizes destructivity over applicability as a real-world attack. Our experiments show that PCFA's applicability in white- and black-box settings, but also show that it finds stronger adversarial samples for optical flow than previous attacking frameworks. We provide the first common ranking of optical flow methods in the literature considering both prediction quality and adversarial robustness, indicating that high quality methods are not necessarily robust.
  arXiv  Detail & Related papers  (2022-03-24T17:10:26Z)
• Benchmarking the Robustness of Spatial-Temporal Models Against Corruptions [32.821121530785504]
  We establish a corruption robustness benchmark, Mini Kinetics-C and Mini SSV2-C, which considers temporal corruptions beyond spatial corruptions in images. We make the first attempt to conduct an exhaustive study on the corruption robustness of established CNN-based and Transformer-based spatial-temporal models.
  arXiv  Detail & Related papers  (2021-10-13T05:59:39Z)
• Sensor-Guided Optical Flow [53.295332513139925]
  This paper proposes a framework to guide an optical flow network with external cues to achieve superior accuracy on known or unseen domains. We show how these can be obtained by combining depth measurements from active sensors with geometry and hand-crafted optical flow algorithms.
  arXiv  Detail & Related papers  (2021-09-30T17:59:57Z)
• Dense Optical Flow from Event Cameras [55.79329250951028]
  We propose to incorporate feature correlation and sequential processing into dense optical flow estimation from event cameras. Our proposed approach computes dense optical flow and reduces the end-point error by 23% on MVSEC.
  arXiv  Detail & Related papers  (2021-08-24T07:39:08Z)
• PCA Event-Based Otical Flow for Visual Odometry [0.0]
  We present a Principal Component Analysis approach to the problem of event-based optical flow estimation. We show that the best variant of our proposed method, dedicated to the real-time context of visual odometry, is about two times faster compared to state-of-the-art implementations.
  arXiv  Detail & Related papers  (2021-05-08T18:30:44Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• What Matters in Unsupervised Optical Flow [51.45112526506455]
  We compare and analyze a set of key components in unsupervised optical flow. We construct a number of novel improvements to unsupervised flow models. We present a new unsupervised flow technique that significantly outperforms the previous state-of-the-art.
  arXiv  Detail & Related papers  (2020-06-08T19:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.