Corruptions of Supervised Learning Problems: Typology and Mitigations

• URL: http://arxiv.org/abs/2307.08643v2
• Date: Thu, 2 May 2024 22:40:09 GMT
• Title: Corruptions of Supervised Learning Problems: Typology and Mitigations
• Authors: Laura Iacovissi, Nan Lu, Robert C. Williamson,
• Abstract summary: We develop a general theory of corruption from an information-theoretic perspective. We will focus here on changes in probability distributions. This work sheds light on complexities arising from joint and dependent corruptions on both labels and attributes.
• Score: 11.294508617469905
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Corruption is notoriously widespread in data collection. Despite extensive research, the existing literature on corruption predominantly focuses on specific settings and learning scenarios, lacking a unified view. There is still a limited understanding of how to effectively model and mitigate corruption in machine learning problems. In this work, we develop a general theory of corruption from an information-theoretic perspective - with Markov kernels as a foundational mathematical tool. We generalize the definition of corruption beyond the concept of distributional shift: corruption includes all modifications of a learning problem, including changes in model class and loss function. We will focus here on changes in probability distributions. First, we construct a provably exhaustive framework for pairwise Markovian corruptions. The framework not only allows us to study corruption types based on their input space, but also serves to unify prior works on specific corruption models and establish a consistent nomenclature. Second, we systematically analyze the consequences of corruption on learning tasks by comparing Bayes risks in the clean and corrupted scenarios. This examination sheds light on complexities arising from joint and dependent corruptions on both labels and attributes. Notably, while label corruptions affect only the loss function, more intricate cases involving attribute corruptions extend the influence beyond the loss to affect the hypothesis class. Third, building upon these results, we investigate mitigations for various corruption types. We expand the existing loss-correction results for label corruption, and identify the necessity to generalize the classical corruption-corrected learning framework to a new paradigm with weaker requirements. Within the latter setting, we provide a negative result for loss correction in the attribute and the joint corruption case.

Related papers

• Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification [17.288347876319126]
  In linear bandits, how can a learner effectively learn when facing corrupted rewards? We compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the action chosen by the learner, and weak corruption, where the corruption level does not depend on the action chosen by the learner. For linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions.
  arXiv  Detail & Related papers  (2024-10-10T02:01:46Z)
• Hierarchical Contrastive Learning for Pattern-Generalizable Image Corruption Detection [40.04083743934034]
  We develop a hierarchical contrastive learning framework to detect corrupted regions. A specialized hierarchical interaction mechanism is designed to facilitate the knowledge of contrastive learning in different scales. Our model has well generalization ability across different corruption patterns.
  arXiv  Detail & Related papers  (2023-08-27T10:03:48Z)
• Frequency-Based Vulnerability Analysis of Deep Learning Models against Image Corruptions [48.34142457385199]
  We present MUFIA, an algorithm designed to identify the specific types of corruptions that can cause models to fail. We find that even state-of-the-art models trained to be robust against known common corruptions struggle against the low visibility-based corruptions crafted by MUFIA.
  arXiv  Detail & Related papers  (2023-06-12T15:19:13Z)
• Point-DAE: Denoising Autoencoders for Self-supervised Point Cloud Learning [54.51061298877896]
  We explore a more general denoising autoencoder for point cloud learning (Point-DAE) by investigating more types of corruptions beyond masking. Specifically, we degrade the point cloud with certain corruptions as input, and learn an encoder-decoder model to reconstruct the original point cloud from its corrupted version.
  arXiv  Detail & Related papers  (2022-11-13T08:02:03Z)
• Soft Diffusion: Score Matching for General Corruptions [84.26037497404195]
  We propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process. We show that our objective learns the gradient of the likelihood under suitable regularity conditions for the family of corruption processes. Our method achieves state-of-the-art FID score $1.85$ on CelebA-64, outperforming all previous linear diffusion models.
  arXiv  Detail & Related papers  (2022-09-12T17:45:03Z)
• Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptions [98.75618795470524]
  We study the linear contextual bandit problem in the presence of adversarial corruption. We propose a new algorithm based on the principle of optimism in the face of uncertainty.
  arXiv  Detail & Related papers  (2022-05-13T17:58:58Z)
• Using the Overlapping Score to Improve Corruption Benchmarks [6.445605125467574]
  We propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. We argue that taking into account overlappings between corruptions can help to improve existing benchmarks or build better ones.
  arXiv  Detail & Related papers  (2021-05-26T06:42:54Z)
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness [78.6626755563546]
  Several new data augmentations have been proposed that significantly improve performance on ImageNet-C. We develop a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate there is a strong correlation between similarity and performance. We observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C. Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may not generalize well beyond the existing benchmark.
  arXiv  Detail & Related papers  (2021-02-22T18:58:39Z)
• Corruption-robust exploration in episodic reinforcement learning [76.19192549843727]
  We study multi-stage episodic reinforcement learning under adversarial corruptions in both the rewards and the transition probabilities of the underlying system. Our framework yields efficient algorithms which attain near-optimal regret in the absence of corruptions. Notably, our work provides the first sublinear regret guarantee which any deviation from purely i.i.d. transitions in the bandit-feedback model for episodic reinforcement learning.
  arXiv  Detail & Related papers  (2019-11-20T03:49:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.