Using the Overlapping Score to Improve Corruption Benchmarks

• URL: http://arxiv.org/abs/2105.12357v1
• Date: Wed, 26 May 2021 06:42:54 GMT
• Title: Using the Overlapping Score to Improve Corruption Benchmarks
• Authors: Alfred Laugros and Alice Caplier and Matthieu Ospici
• Abstract summary: We propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. We argue that taking into account overlappings between corruptions can help to improve existing benchmarks or build better ones.
• Score: 6.445605125467574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Networks are sensitive to various corruptions that usually occur in real-world applications such as blurs, noises, low-lighting conditions, etc. To estimate the robustness of neural networks to these common corruptions, we generally use a group of modeled corruptions gathered into a benchmark. Unfortunately, no objective criterion exists to determine whether a benchmark is representative of a large diversity of independent corruptions. In this paper, we propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. Two corruptions overlap when the robustnesses of neural networks to these corruptions are correlated. We argue that taking into account overlappings between corruptions can help to improve existing benchmarks or build better ones.

Related papers

• Assessing Visually-Continuous Corruption Robustness of Neural Networks Relative to Human Performance [6.254768374567899]
  Neural Networks (NNs) have surpassed human accuracy in image classification on ImageNet. NNs often lack robustness against image corruption, i.e., corruption robustness. We propose visually-continuous corruption robustness (VCR) to allow assessing it over the wide and continuous range of changes that correspond to the human perceptive quality.
  arXiv  Detail & Related papers  (2024-02-29T18:00:27Z)
• Enhancing Infrared Small Target Detection Robustness with Bi-Level Adversarial Framework [61.34862133870934]
  We propose a bi-level adversarial framework to promote the robustness of detection in the presence of distinct corruptions. Our scheme remarkably improves 21.96% IOU across a wide array of corruptions and notably promotes 4.97% IOU on the general benchmark.
  arXiv  Detail & Related papers  (2023-09-03T06:35:07Z)
• Corruptions of Supervised Learning Problems: Typology and Mitigations [11.294508617469905]
  We develop a general theory of corruption from an information-theoretic perspective. We will focus here on changes in probability distributions. This work sheds light on complexities arising from joint and dependent corruptions on both labels and attributes.
  arXiv  Detail & Related papers  (2023-07-17T16:57:01Z)
• Frequency-Based Vulnerability Analysis of Deep Learning Models against Image Corruptions [48.34142457385199]
  We present MUFIA, an algorithm designed to identify the specific types of corruptions that can cause models to fail. We find that even state-of-the-art models trained to be robust against known common corruptions struggle against the low visibility-based corruptions crafted by MUFIA.
  arXiv  Detail & Related papers  (2023-06-12T15:19:13Z)
• Investigating the Corruption Robustness of Image Classifiers with Random Lp-norm Corruptions [3.1337872355726084]
  This study investigates the use of random p-norm corruptions to augment the training and test data of image classifiers. We find that training data augmentation with a combination of p-norm corruptions significantly improves corruption robustness, even on top of state-of-the-art data augmentation schemes.
  arXiv  Detail & Related papers  (2023-05-09T12:45:43Z)
• Soft Diffusion: Score Matching for General Corruptions [84.26037497404195]
  We propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process. We show that our objective learns the gradient of the likelihood under suitable regularity conditions for the family of corruption processes. Our method achieves state-of-the-art FID score $1.85$ on CelebA-64, outperforming all previous linear diffusion models.
  arXiv  Detail & Related papers  (2022-09-12T17:45:03Z)
• Benchmarking the Robustness of Deep Neural Networks to Common Corruptions in Digital Pathology [11.398235052118608]
  This benchmark is established to evaluate how deep neural networks perform on corrupted pathology images. Two classification and one ranking metrics are designed to evaluate the prediction and confidence performance under corruption.
  arXiv  Detail & Related papers  (2022-06-30T01:53:46Z)
• Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptions [98.75618795470524]
  We study the linear contextual bandit problem in the presence of adversarial corruption. We propose a new algorithm based on the principle of optimism in the face of uncertainty.
  arXiv  Detail & Related papers  (2022-05-13T17:58:58Z)
• Linear Contextual Bandits with Adversarial Corruptions [91.38793800392108]
  We study the linear contextual bandit problem in the presence of adversarial corruption. We present a variance-aware algorithm that is adaptive to the level of adversarial contamination $C$.
  arXiv  Detail & Related papers  (2021-10-25T02:53:24Z)
• Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts [6.445605125467574]
  We propose a methodology to build synthetic corruption benchmarks that make robustness estimations more correlated with robustness to real-world distribution shifts. Applying the proposed methodology, we build a new benchmark called ImageNet-Syn2Nat to predict image classifier robustness.
  arXiv  Detail & Related papers  (2021-07-26T09:20:49Z)
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness [78.6626755563546]
  Several new data augmentations have been proposed that significantly improve performance on ImageNet-C. We develop a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate there is a strong correlation between similarity and performance. We observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may not generalize well beyond the existing benchmark.
  arXiv  Detail & Related papers  (2021-02-22T18:58:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.