Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure

• URL: http://arxiv.org/abs/2301.04850v1
• Date: Thu, 12 Jan 2023 07:28:32 GMT
• Title: Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure
• Authors: Xiaoling Zhou, Ou Wu, Weiyao Zhu, Ziyang Liang
• Abstract summary: A large number of weighting methods essentially utilize the learning difficulty of training samples to calculate their weights. The learning difficulties of the samples are determined by multiple factors including noise level, imbalance degree, margin, and uncertainty. In this study, we theoretically prove that the generalization error of a sample can be used as a universal difficulty measure.
• Score: 2.7413469516930578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sample weighting is widely used in deep learning. A large number of weighting methods essentially utilize the learning difficulty of training samples to calculate their weights. In this study, this scheme is called difficulty-based weighting. Two important issues arise when explaining this scheme. First, a unified difficulty measure that can be theoretically guaranteed for training samples does not exist. The learning difficulties of the samples are determined by multiple factors including noise level, imbalance degree, margin, and uncertainty. Nevertheless, existing measures only consider a single factor or in part, but not in their entirety. Second, a comprehensive theoretical explanation is lacking with respect to demonstrating why difficulty-based weighting schemes are effective in deep learning. In this study, we theoretically prove that the generalization error of a sample can be used as a universal difficulty measure. Furthermore, we provide formal theoretical justifications on the role of difficulty-based weighting for deep learning, consequently revealing its positive influences on both the optimization dynamics and generalization performance of deep models, which is instructive to existing weighting schemes.

Related papers

• A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning [129.63326990812234]
  We propose a technique named data-dependent contraction to capture how modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
  arXiv  Detail & Related papers  (2023-10-07T09:15:08Z)
• Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs [71.56345106591789]
  It has been believed that weights in large language models (LLMs) contain significant redundancy. This paper presents a counter-argument: small-magnitude weights of pre-trained model weights encode vital knowledge essential for tackling difficult downstream tasks.
  arXiv  Detail & Related papers  (2023-09-29T22:55:06Z)
• Exploring Weight Balancing on Long-Tailed Recognition Problem [32.01426831450348]
  Recognition problems in long-tailed data, in which the sample size per class is heavily skewed, have gained importance. Weight balancing, which combines classical regularization techniques with two-stage training, has been proposed. We analyze weight balancing by focusing on neural collapse and the cone effect at each training stage.
  arXiv  Detail & Related papers  (2023-05-26T01:45:19Z)
• Prototype-Anchored Learning for Learning with Imperfect Annotations [83.7763875464011]
  It is challenging to learn unbiased classification models from imperfectly annotated datasets. We propose a prototype-anchored learning (PAL) method, which can be easily incorporated into various learning-based classification schemes. We verify the effectiveness of PAL on class-imbalanced learning and noise-tolerant learning by extensive experiments on synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-06-23T10:25:37Z)
• Exploring the Learning Difficulty of Data Theory and Measure [2.668651175000491]
  This study attempts to conduct a pilot theoretical study for learning difficulty of samples. A theoretical definition of learning difficulty is proposed on the basis of the bias-variance trade-off theory on generalization error. Several classical weighting methods in machine learning can be well explained on account of explored properties.
  arXiv  Detail & Related papers  (2022-05-16T02:28:12Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• On the Impact of Hard Adversarial Instances on Overfitting in Adversarial Training [72.95029777394186]
  Adversarial training is a popular method to robustify models against adversarial attacks. We investigate this phenomenon from the perspective of training instances. We show that the decay in generalization performance of adversarial training is a result of the model's attempt to fit hard adversarial instances.
  arXiv  Detail & Related papers  (2021-12-14T12:19:24Z)
• Which Samples Should be Learned First: Easy or Hard? [5.589137389571604]
  weighting scheme for training samples is essential for learning tasks. Some schemes take the easy-first mode on samples, whereas some others take the hard-first mode. Factors including prior knowledge and data characteristics determine which samples should be learned first in a learning task.
  arXiv  Detail & Related papers  (2021-10-11T03:40:29Z)
• Probably Approximately Correct Constrained Learning [135.48447120228658]
  We develop a generalization theory based on the probably approximately correct (PAC) learning framework. We show that imposing a learner does not make a learning problem harder in the sense that any PAC learnable class is also a constrained learner. We analyze the properties of this solution and use it to illustrate how constrained learning can address problems in fair and robust classification.
  arXiv  Detail & Related papers  (2020-06-09T19:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.