DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions

• URL: http://arxiv.org/abs/2505.05091v1
• Date: Thu, 08 May 2025 09:40:17 GMT
• Title: DispBench: Benchmarking Disparity Estimation to Synthetic Corruptions
• Authors: Shashank Agnihotri, Amaan Ansari, Annika Dackermann, Fabian Rösch, Margret Keuper,
• Abstract summary: Deep learning (DL) has surpassed human performance on standard benchmarks, driving its widespread adoption in computer vision tasks.<n>DispBench is a comprehensive benchmarking tool for systematically assessing the reliability of disparity estimation methods.<n>We conduct the most extensive performance and robustness analysis of disparity estimation methods to date, uncovering key correlations between accuracy, reliability, and generalization.
• Score: 11.35081321966394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning (DL) has surpassed human performance on standard benchmarks, driving its widespread adoption in computer vision tasks. One such task is disparity estimation, estimating the disparity between matching pixels in stereo image pairs, which is crucial for safety-critical applications like medical surgeries and autonomous navigation. However, DL-based disparity estimation methods are highly susceptible to distribution shifts and adversarial attacks, raising concerns about their reliability and generalization. Despite these concerns, a standardized benchmark for evaluating the robustness of disparity estimation methods remains absent, hindering progress in the field. To address this gap, we introduce DispBench, a comprehensive benchmarking tool for systematically assessing the reliability of disparity estimation methods. DispBench evaluates robustness against synthetic image corruptions such as adversarial attacks and out-of-distribution shifts caused by 2D Common Corruptions across multiple datasets and diverse corruption scenarios. We conduct the most extensive performance and robustness analysis of disparity estimation methods to date, uncovering key correlations between accuracy, reliability, and generalization. Open-source code for DispBench: https://github.com/shashankskagnihotri/benchmarking_robustness/tree/disparity_estimation/final/dispa rity_estimation

Related papers

• Rethinking Robustness in Machine Learning: A Posterior Agreement Approach [45.284633306624634]
  Posterior Agreement (PA) theory of model validation provides a principled framework for robustness evaluation.<n>We show that the PA metric provides a sensible and consistent analysis of the vulnerabilities in learning algorithms, even in the presence of few observations.
  arXiv  Detail & Related papers  (2025-03-20T16:03:39Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Framework [8.572441599469597]
  We study high-confidence off-policy evaluation in the context of infinite-horizon Markov decision processes. The objective is to establish a confidence interval (CI) for the target policy value using only offline data pre-collected from unknown behavior policies. We show that our algorithm is sample-efficient, error-robust, and provably convergent even in non-linear function approximation settings.
  arXiv  Detail & Related papers  (2023-09-23T06:35:44Z)
• Utilizing Class Separation Distance for the Evaluation of Corruption Robustness of Machine Learning Classifiers [0.6882042556551611]
  We propose a test data augmentation method that uses a robustness distance $epsilon$ derived from the datasets minimal class separation distance. The resulting MSCR metric allows a dataset-specific comparison of different classifiers with respect to their corruption robustness. Our results indicate that robustness training through simple data augmentation can already slightly improve accuracy.
  arXiv  Detail & Related papers  (2022-06-27T15:56:16Z)
• Probabilistically Robust Learning: Balancing Average- and Worst-case Performance [105.87195436925722]
  We propose a framework called robustness probabilistic that bridges the gap between the accurate, yet brittle average case and the robust, yet conservative worst case. From a theoretical point of view, this framework overcomes the trade-offs between the performance and the sample-complexity of worst-case and average-case learning.
  arXiv  Detail & Related papers  (2022-02-02T17:01:38Z)
• Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts [6.445605125467574]
  We propose a methodology to build synthetic corruption benchmarks that make robustness estimations more correlated with robustness to real-world distribution shifts. Applying the proposed methodology, we build a new benchmark called ImageNet-Syn2Nat to predict image classifier robustness.
  arXiv  Detail & Related papers  (2021-07-26T09:20:49Z)
• Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation [74.05906222376608]
  We propose adversarial self-supervision UDA (or ASSUDA) that maximizes the agreement between clean images and their adversarial examples by a contrastive loss in the output space. This paper is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, which pose a security concern in this field; and (ii) although commonly used self-supervision (e.g., rotation and jigsaw) benefits image tasks such as classification and recognition, they fail to provide the critical supervision signals that could learn discriminative representation for segmentation tasks.
  arXiv  Detail & Related papers  (2021-05-23T01:50:44Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• RobustBench: a standardized adversarial robustness benchmark [84.50044645539305]
  Key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks. We analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.
  arXiv  Detail & Related papers  (2020-10-19T17:06:18Z)
• Robust Learning Through Cross-Task Consistency [92.42534246652062]
  We propose a broadly applicable and fully computational method for augmenting learning with Cross-Task Consistency. We observe that learning with cross-task consistency leads to more accurate predictions and better generalization to out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-06-07T09:24:33Z)
• On Robust Mean Estimation under Coordinate-level Corruption [23.117927954549618]
  We introduce a novel Hamming distance-based measure of distribution shift for coordinate-level corruptions. We show that this measure yields adversary models that capture more realistic corruptions than those used in prior works. We show that for structured distributions, methods that leverage the structure yield information theoretically more accurate mean estimation.
  arXiv  Detail & Related papers  (2020-02-10T23:48:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.