Unified Regularity Measures for Sample-wise Learning and Generalization
- URL: http://arxiv.org/abs/2108.03913v1
- Date: Mon, 9 Aug 2021 10:11:14 GMT
- Title: Unified Regularity Measures for Sample-wise Learning and Generalization
- Authors: Chi Zhang, Xiaoning Ma, Yu Liu, Le Wang, Yuanqi Su, Yuehu Liu
- Abstract summary: We propose a pair of sample regularity measures for both processes with a formulation-consistent representation.
Experiments validated the effectiveness and robustness of the proposed approaches for mini-batch SGD optimization.
- Score: 18.10522585996242
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Fundamental machine learning theory shows that different samples contribute
unequally both in learning and testing processes. Contemporary studies on DNN
imply that such sample di?erence is rooted on the distribution of intrinsic
pattern information, namely sample regularity. Motivated by the recent
discovery on network memorization and generalization, we proposed a pair of
sample regularity measures for both processes with a formulation-consistent
representation. Specifically, cumulative binary training/generalizing loss
(CBTL/CBGL), the cumulative number of correct classi?cations of the
training/testing sample within training stage, is proposed to quantize the
stability in memorization-generalization process; while
forgetting/mal-generalizing events, i.e., the mis-classification of previously
learned or generalized sample, are utilized to represent the uncertainty of
sample regularity with respect to optimization dynamics. Experiments validated
the effectiveness and robustness of the proposed approaches for mini-batch SGD
optimization. Further applications on training/testing sample selection show
the proposed measures sharing the uni?ed computing procedure could benefit for
both tasks.
Related papers
- Adaptive Intra-Class Variation Contrastive Learning for Unsupervised Person Re-Identification [10.180143197144803]
We propose an adaptive intra-class variation contrastive learning algorithm for unsupervised Re-ID, called AdaInCV.
The algorithm quantitatively evaluates the learning ability of the model for each class by considering the intra-class variations after clustering.
To be more specific, two new strategies are proposed: Adaptive Sample Mining (AdaSaM) and Adaptive Outlier Filter (AdaOF)
arXiv Detail & Related papers (2024-04-06T15:48:14Z) - Simple and effective data augmentation for compositional generalization [64.00420578048855]
We show that data augmentation methods that sample MRs and backtranslate them can be effective for compositional generalization.
Remarkably, sampling from a uniform distribution performs almost as well as sampling from the test distribution.
arXiv Detail & Related papers (2024-01-18T09:13:59Z) - Learning Less Generalizable Patterns with an Asymmetrically Trained
Double Classifier for Better Test-Time Adaptation [4.893694715581673]
We propose a novel approach using a pair of classifiers and a shortcut patterns avoidance loss.
Our method improves upon the state-of-the-art results on both benchmarks and benefits the most to test-time batch normalization.
arXiv Detail & Related papers (2022-10-17T08:05:38Z) - Jo-SRC: A Contrastive Approach for Combating Noisy Labels [58.867237220886885]
We propose a noise-robust approach named Jo-SRC (Joint Sample Selection and Model Regularization based on Consistency)
Specifically, we train the network in a contrastive learning manner. Predictions from two different views of each sample are used to estimate its "likelihood" of being clean or out-of-distribution.
arXiv Detail & Related papers (2021-03-24T07:26:07Z) - Squared $\ell_2$ Norm as Consistency Loss for Leveraging Augmented Data
to Learn Robust and Invariant Representations [76.85274970052762]
Regularizing distance between embeddings/representations of original samples and augmented counterparts is a popular technique for improving robustness of neural networks.
In this paper, we explore these various regularization choices, seeking to provide a general understanding of how we should regularize the embeddings.
We show that the generic approach we identified (squared $ell$ regularized augmentation) outperforms several recent methods, which are each specially designed for one task.
arXiv Detail & Related papers (2020-11-25T22:40:09Z) - Optimal Importance Sampling for Federated Learning [57.14673504239551]
Federated learning involves a mixture of centralized and decentralized processing tasks.
The sampling of both agents and data is generally uniform; however, in this work we consider non-uniform sampling.
We derive optimal importance sampling strategies for both agent and data selection and show that non-uniform sampling without replacement improves the performance of the original FedAvg algorithm.
arXiv Detail & Related papers (2020-10-26T14:15:33Z) - A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution.
We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
arXiv Detail & Related papers (2020-07-08T11:35:47Z) - Robust Sampling in Deep Learning [62.997667081978825]
Deep learning requires regularization mechanisms to reduce overfitting and improve generalization.
We address this problem by a new regularization method based on distributional robust optimization.
During the training, the selection of samples is done according to their accuracy in such a way that the worst performed samples are the ones that contribute the most in the optimization.
arXiv Detail & Related papers (2020-06-04T09:46:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.