Realistic Evaluation of Transductive Few-Shot Learning
- URL: http://arxiv.org/abs/2204.11181v1
- Date: Sun, 24 Apr 2022 03:35:06 GMT
- Title: Realistic Evaluation of Transductive Few-Shot Learning
- Authors: Olivier Veilleux, Malik Boudiaf, Pablo Piantanida, Ismail Ben Ayed
- Abstract summary: Transductive inference is widely used in few-shot learning.
We study the effect of arbitrary class distributions within the query sets of few-shot tasks at inference.
We evaluate experimentally state-of-the-art transductive methods over 3 widely used data sets.
- Score: 41.06192162435249
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Transductive inference is widely used in few-shot learning, as it leverages
the statistics of the unlabeled query set of a few-shot task, typically
yielding substantially better performances than its inductive counterpart. The
current few-shot benchmarks use perfectly class-balanced tasks at inference. We
argue that such an artificial regularity is unrealistic, as it assumes that the
marginal label probability of the testing samples is known and fixed to the
uniform distribution. In fact, in realistic scenarios, the unlabeled query sets
come with arbitrary and unknown label marginals. We introduce and study the
effect of arbitrary class distributions within the query sets of few-shot tasks
at inference, removing the class-balance artefact. Specifically, we model the
marginal probabilities of the classes as Dirichlet-distributed random
variables, which yields a principled and realistic sampling within the simplex.
This leverages the current few-shot benchmarks, building testing tasks with
arbitrary class distributions. We evaluate experimentally state-of-the-art
transductive methods over 3 widely used data sets, and observe, surprisingly,
substantial performance drops, even below inductive methods in some cases.
Furthermore, we propose a generalization of the mutual-information loss, based
on $\alpha$-divergences, which can handle effectively class-distribution
variations. Empirically, we show that our transductive $\alpha$-divergence
optimization outperforms state-of-the-art methods across several data sets,
models and few-shot settings. Our code is publicly available at
https://github.com/oveilleux/Realistic_Transductive_Few_Shot.
Related papers
- SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning [49.94607673097326]
We propose a highly adaptable framework, designated as SimPro, which does not rely on any predefined assumptions about the distribution of unlabeled data.
Our framework, grounded in a probabilistic model, innovatively refines the expectation-maximization algorithm.
Our method showcases consistent state-of-the-art performance across diverse benchmarks and data distribution scenarios.
arXiv Detail & Related papers (2024-02-21T03:39:04Z) - Dr. FERMI: A Stochastic Distributionally Robust Fair Empirical Risk
Minimization Framework [12.734559823650887]
In the presence of distribution shifts, fair machine learning models may behave unfairly on test data.
Existing algorithms require full access to data and cannot be used when small batches are used.
This paper proposes the first distributionally robust fairness framework with convergence guarantees that do not require knowledge of the causal graph.
arXiv Detail & Related papers (2023-09-20T23:25:28Z) - Adaptive manifold for imbalanced transductive few-shot learning [16.627512688664513]
We propose a novel algorithm to address imbalanced transductive few-shot learning, named Adaptive Manifold.
Our method exploits the underlying manifold of the labeled support examples and unlabeled queries by using manifold similarity to predict the class probability distribution per query.
arXiv Detail & Related papers (2023-04-27T15:42:49Z) - Accounting for multiplicity in machine learning benchmark performance [0.0]
Using the highest-ranked performance as an estimate for state-of-the-art (SOTA) performance is a biased estimator, giving overly optimistic results.
In this article, we provide a probability distribution for the case of multiple classifiers so that known analyses methods can be engaged and a better SOTA estimate can be provided.
arXiv Detail & Related papers (2023-03-10T10:32:18Z) - An Upper Bound for the Distribution Overlap Index and Its Applications [18.481370450591317]
This paper proposes an easy-to-compute upper bound for the overlap index between two probability distributions.
The proposed bound shows its value in one-class classification and domain shift analysis.
Our work shows significant promise toward broadening the applications of overlap-based metrics.
arXiv Detail & Related papers (2022-12-16T20:02:03Z) - Predicting Out-of-Domain Generalization with Neighborhood Invariance [59.05399533508682]
We propose a measure of a classifier's output invariance in a local transformation neighborhood.
Our measure is simple to calculate, does not depend on the test point's true label, and can be applied even in out-of-domain (OOD) settings.
In experiments on benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our measure and actual OOD generalization.
arXiv Detail & Related papers (2022-07-05T14:55:16Z) - Smoothed Embeddings for Certified Few-Shot Learning [63.68667303948808]
We extend randomized smoothing to few-shot learning models that map inputs to normalized embeddings.
Our results are confirmed by experiments on different datasets.
arXiv Detail & Related papers (2022-02-02T18:19:04Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z) - Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
Machine learning algorithms are susceptible to data poisoning attacks.
We present a unifying view of randomized smoothing over arbitrary functions.
We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
arXiv Detail & Related papers (2020-02-07T21:28:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.