A Study on the Predictability of Sample Learning Consistency
- URL: http://arxiv.org/abs/2207.03571v1
- Date: Thu, 7 Jul 2022 21:05:53 GMT
- Title: A Study on the Predictability of Sample Learning Consistency
- Authors: Alain Raymond-Saez, Julio Hurtado, Alvaro Soto
- Abstract summary: We train models to predict C-Score for CIFAR-100 and CIFAR-10.
We find, however, that these models generalize poorly both within the same distribution as well as out of distribution.
We hypothesize that a sample's relation to its neighbours, in particular, how many of them share the same labels, can help in explaining C-Scores.
- Score: 4.022364531869169
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Curriculum Learning is a powerful training method that allows for faster and
better training in some settings. This method, however, requires having a
notion of which examples are difficult and which are easy, which is not always
trivial to provide. A recent metric called C-Score acts as a proxy for example
difficulty by relating it to learning consistency. Unfortunately, this method
is quite compute intensive which limits its applicability for alternative
datasets. In this work, we train models through different methods to predict
C-Score for CIFAR-100 and CIFAR-10. We find, however, that these models
generalize poorly both within the same distribution as well as out of
distribution. This suggests that C-Score is not defined by the individual
characteristics of each sample but rather by other factors. We hypothesize that
a sample's relation to its neighbours, in particular, how many of them share
the same labels, can help in explaining C-Scores. We plan to explore this in
future work.
Related papers
- Probably Approximately Precision and Recall Learning [62.912015491907994]
Precision and Recall are foundational metrics in machine learning.
One-sided feedback--where only positive examples are observed during training--is inherent in many practical problems.
We introduce a PAC learning framework where each hypothesis is represented by a graph, with edges indicating positive interactions.
arXiv Detail & Related papers (2024-11-20T04:21:07Z) - FeCAM: Exploiting the Heterogeneity of Class Distributions in
Exemplar-Free Continual Learning [21.088762527081883]
Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks.
Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention.
We explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes.
arXiv Detail & Related papers (2023-09-25T11:54:33Z) - Intra-class Adaptive Augmentation with Neighbor Correction for Deep
Metric Learning [99.14132861655223]
We propose a novel intra-class adaptive augmentation (IAA) framework for deep metric learning.
We reasonably estimate intra-class variations for every class and generate adaptive synthetic samples to support hard samples mining.
Our method significantly improves and outperforms the state-of-the-art methods on retrieval performances by 3%-6%.
arXiv Detail & Related papers (2022-11-29T14:52:38Z) - Bias Mimicking: A Simple Sampling Approach for Bias Mitigation [57.17709477668213]
We introduce a new class-conditioned sampling method: Bias Mimicking.
Bias Mimicking improves underrepresented groups' accuracy of sampling methods by 3% over four benchmarks.
arXiv Detail & Related papers (2022-09-30T17:33:00Z) - Learning with Neighbor Consistency for Noisy Labels [69.83857578836769]
We present a method for learning from noisy labels that leverages similarities between training examples in feature space.
We evaluate our method on datasets evaluating both synthetic (CIFAR-10, CIFAR-100) and realistic (mini-WebVision, Clothing1M, mini-ImageNet-Red) noise.
arXiv Detail & Related papers (2022-02-04T15:46:27Z) - An Empirical Comparison of Instance Attribution Methods for NLP [62.63504976810927]
We evaluate the degree to which different potential instance attribution agree with respect to the importance of training samples.
We find that simple retrieval methods yield training instances that differ from those identified via gradient-based methods.
arXiv Detail & Related papers (2021-04-09T01:03:17Z) - Contrastive learning of strong-mixing continuous-time stochastic
processes [53.82893653745542]
Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data.
We show that a properly constructed contrastive learning task can be used to estimate the transition kernel for small-to-mid-range intervals in the diffusion case.
arXiv Detail & Related papers (2021-03-03T23:06:47Z) - Classification with Strategically Withheld Data [41.78264347024645]
Machine learning techniques can be useful in applications such as credit approval and college admission.
To be classified more favorably in such contexts, an agent may decide to strategically withhold some of her features, such as bad test scores.
We design three classification methods: sc Mincut, sc Hill-Climbing (sc HC) and Incentive- Logistic Regression (sc-LR)
arXiv Detail & Related papers (2020-12-18T12:54:41Z) - An end-to-end approach for the verification problem: learning the right
distance [15.553424028461885]
We augment the metric learning setting by introducing a parametric pseudo-distance, trained jointly with the encoder.
We first show it approximates a likelihood ratio which can be used for hypothesis tests.
We observe training is much simplified under the proposed approach compared to metric learning with actual distances.
arXiv Detail & Related papers (2020-02-21T18:46:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.