DeepMoM: Robust Deep Learning With Median-of-Means

• URL: http://arxiv.org/abs/2105.14035v1
• Date: Fri, 28 May 2021 18:07:32 GMT
• Title: DeepMoM: Robust Deep Learning With Median-of-Means
• Authors: Shih-Ting Huang and Johannes Lederer
• Abstract summary: We introduce an approach motivated by very recent insights into median-of-means and Le Cam's principle. We show that the approach can be readily implemented, and we demonstrate that it performs very well in practice.
• Score: 0.3553493344868413
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Data used in deep learning is notoriously problematic. For example, data are usually combined from diverse sources, rarely cleaned and vetted thoroughly, and sometimes corrupted on purpose. Intentional corruption that targets the weak spots of algorithms has been studied extensively under the label of "adversarial attacks." In contrast, the arguably much more common case of corruption that reflects the limited quality of data has been studied much less. Such "random" corruptions are due to measurement errors, unreliable sources, convenience sampling, and so forth. These kinds of corruption are common in deep learning, because data are rarely collected according to strict protocols -- in strong contrast to the formalized data collection in some parts of classical statistics. This paper concerns such corruption. We introduce an approach motivated by very recent insights into median-of-means and Le Cam's principle, we show that the approach can be readily implemented, and we demonstrate that it performs very well in practice. In conclusion, we believe that our approach is a very promising alternative to standard parameter training based on least-squares and cross-entropy loss.

Related papers

• Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information [55.75102049412629]
  We show that effective unlearnable examples always decrease mutual information between clean features and poisoned features.<n>We propose a novel unlearnable method called Mutual Information Unlearnable Examples (MI-UE)<n>Our approach significantly outperforms the previous methods, even under defense mechanisms.
  arXiv  Detail & Related papers  (2026-03-04T04:53:29Z)
• Online Learning to Rank under Corruption: A Robust Cascading Bandits Approach [15.847551488328866]
  Online learning to rank studies how to recommend a short ranked list of items from a large pool and improves future rankings based on user clicks.<n>This setting is commonly modeled as cascading bandits, where the objective is to maximize the likelihood that the user clicks on at least one of the presented items.<n>We propose MSUCB, a robust algorithm that incorporates a novel mean-of-medians estimator, which is applied to bandits with corruption setting.
  arXiv  Detail & Related papers  (2025-11-04T23:39:37Z)
• Linear Regression under Missing or Corrupted Coordinates [58.9213131489513]
  We study how data may be corrupted or erased by an adversary under a coordinate-wise budget.<n>In the incomplete data setting, an adversary may inspect the dataset and delete entries in up to an $eta$-fraction of samples per coordinate.<n>In the corrupted data setting, the adversary instead replaces values arbitrarily, and the corruption locations are unknown to the learner.
  arXiv  Detail & Related papers  (2025-09-23T17:01:43Z)
• Geometric Median Matching for Robust k-Subset Selection from Noisy Data [75.86423267723728]
  We propose a novel k-subset selection strategy that leverages Geometric Median -- a robust estimator with an optimal breakdown point of 1/2. Our method iteratively selects a k-subset such that the mean of the subset approximates the GM of the (potentially) noisy dataset, ensuring robustness even under arbitrary corruption.
  arXiv  Detail & Related papers  (2025-04-01T09:22:05Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Late Stopping: Avoiding Confidently Learning from Mislabeled Examples [61.00103151680946]
  We propose a new framework, Late Stopping, which leverages the intrinsic robust learning ability of DNNs through a prolonged training process. We empirically observe that mislabeled and clean examples exhibit differences in the number of epochs required for them to be consistently and correctly classified. Experimental results on benchmark-simulated and real-world noisy datasets demonstrate that the proposed method outperforms state-of-the-art counterparts.
  arXiv  Detail & Related papers  (2023-08-26T12:43:25Z)
• Frequency-Based Vulnerability Analysis of Deep Learning Models against Image Corruptions [48.34142457385199]
  We present MUFIA, an algorithm designed to identify the specific types of corruptions that can cause models to fail. We find that even state-of-the-art models trained to be robust against known common corruptions struggle against the low visibility-based corruptions crafted by MUFIA.
  arXiv  Detail & Related papers  (2023-06-12T15:19:13Z)
• Soft Diffusion: Score Matching for General Corruptions [84.26037497404195]
  We propose a new objective called Soft Score Matching that provably learns the score function for any linear corruption process. We show that our objective learns the gradient of the likelihood under suitable regularity conditions for the family of corruption processes. Our method achieves state-of-the-art FID score $1.85$ on CelebA-64, outperforming all previous linear diffusion models.
  arXiv  Detail & Related papers  (2022-09-12T17:45:03Z)
• Robust estimation algorithms don't need to know the corruption level [50.31562134370949]
  Robust estimation algorithms can perform well even when part of the data is corrupt. Their vast majority approach optimal accuracy only when given a tight upper bound on the fraction of corrupt data. This brief note abstracts the complex and pervasive robustness problem into a simple geometric puzzle. It applies the puzzle's solution to derive a universal meta technique.
  arXiv  Detail & Related papers  (2022-02-11T05:18:28Z)
• Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning [57.88785630755165]
  Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data. We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class. For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
  arXiv  Detail & Related papers  (2021-06-03T09:50:13Z)
• Using the Overlapping Score to Improve Corruption Benchmarks [6.445605125467574]
  We propose a metric called corruption overlapping score, which can be used to reveal flaws in corruption benchmarks. We argue that taking into account overlappings between corruptions can help to improve existing benchmarks or build better ones.
  arXiv  Detail & Related papers  (2021-05-26T06:42:54Z)
• Towards Error Measures which Influence a Learners Inductive Bias to the Ground Truth [0.0]
  This paper investigates how error measures affect the ability for a regression method to model the ground truth' in scenarios with sparse data. Current error measures are shown to create an unhelpful bias and a new error measure is derived which does not exhibit this behaviour.
  arXiv  Detail & Related papers  (2021-05-04T15:33:58Z)
• Are Bias Mitigation Techniques for Deep Learning Effective? [24.84797949716142]
  We introduce an improved evaluation protocol, sensible metrics, and a new dataset. We evaluate seven state-of-the-art algorithms using the same network architecture. We find that algorithms exploit hidden biases, are unable to scale to multiple forms of bias, and are highly sensitive to the choice of tuning set.
  arXiv  Detail & Related papers  (2021-04-01T00:14:45Z)
• On Provable Backdoor Defense in Collaborative Learning [35.22450536986004]
  Malicious users can upload data to prevent the model's convergence or inject hidden backdoors. Backdoor attacks are especially difficult to detect since the model behaves normally on standard test data but gives wrong outputs when triggered by certain backdoor keys. We propose a novel framework that generalizes existing subset aggregation methods.
  arXiv  Detail & Related papers  (2021-01-19T14:39:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.