On the Role of Neural Collapse in Transfer Learning
- URL: http://arxiv.org/abs/2112.15121v1
- Date: Thu, 30 Dec 2021 16:36:26 GMT
- Title: On the Role of Neural Collapse in Transfer Learning
- Authors: Tomer Galanti, Andr\'as Gy\"orgy, Marcus Hutter
- Abstract summary: Recent results show that representations learned by a single classifier over many classes are competitive on few-shot learning problems.
We show that neural collapse generalizes to new samples from the training classes, and -- more importantly -- to new classes as well.
- Score: 29.972063833424215
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We study the ability of foundation models to learn representations for
classification that are transferable to new, unseen classes. Recent results in
the literature show that representations learned by a single classifier over
many classes are competitive on few-shot learning problems with representations
learned by special-purpose algorithms designed for such problems. In this paper
we provide an explanation for this behavior based on the recently observed
phenomenon that the features learned by overparameterized classification
networks show an interesting clustering property, called neural collapse. We
demonstrate both theoretically and empirically that neural collapse generalizes
to new samples from the training classes, and -- more importantly -- to new
classes as well, allowing foundation models to provide feature maps that work
well in transfer learning and, specifically, in the few-shot setting.
Related papers
- Few-Shot Class-Incremental Learning via Training-Free Prototype
Calibration [67.69532794049445]
We find a tendency for existing methods to misclassify the samples of new classes into base classes, which leads to the poor performance of new classes.
We propose a simple yet effective Training-frEE calibratioN (TEEN) strategy to enhance the discriminability of new classes.
arXiv Detail & Related papers (2023-12-08T18:24:08Z) - Generalized Neural Collapse for a Large Number of Classes [33.46269920297418]
We provide empirical study to verify the occurrence of generalized neural collapse in practical deep neural networks.
We provide theoretical study to show that the generalized neural collapse provably occurs under unconstrained feature model with spherical constraint.
arXiv Detail & Related papers (2023-10-09T02:27:04Z) - On the Role of Neural Collapse in Meta Learning Models for Few-shot
Learning [0.9729803206187322]
This study is the first to explore and understand the properties of neural collapse in meta learning frameworks for few-shot learning.
We perform studies on the Omniglot dataset in the few-shot setting and study the neural collapse phenomenon.
arXiv Detail & Related papers (2023-09-30T18:02:51Z) - Generalization Bounds for Few-Shot Transfer Learning with Pretrained
Classifiers [26.844410679685424]
We study the ability of foundation models to learn representations for classification that are transferable to new, unseen classes.
We show that the few-shot error of the learned feature map on new classes is small in case of class-feature-variability collapse.
arXiv Detail & Related papers (2022-12-23T18:46:05Z) - Continual Learning with Bayesian Model based on a Fixed Pre-trained
Feature Extractor [55.9023096444383]
Current deep learning models are characterised by catastrophic forgetting of old knowledge when learning new classes.
Inspired by the process of learning new knowledge in human brains, we propose a Bayesian generative model for continual learning.
arXiv Detail & Related papers (2022-04-28T08:41:51Z) - Learning Debiased and Disentangled Representations for Semantic
Segmentation [52.35766945827972]
We propose a model-agnostic and training scheme for semantic segmentation.
By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes.
Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
arXiv Detail & Related papers (2021-10-31T16:15:09Z) - Adversarially-Trained Deep Nets Transfer Better: Illustration on Image
Classification [53.735029033681435]
Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains.
In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
arXiv Detail & Related papers (2020-07-11T22:48:42Z) - Looking back to lower-level information in few-shot learning [4.873362301533825]
We propose the utilization of lower-level, supporting information, namely the feature embeddings of the hidden neural network layers, to improve classification accuracy.
Our experiments on two popular few-shot learning datasets, miniImageNet and tieredImageNet, show that our method can utilize the lower-level information in the network to improve state-of-the-art classification performance.
arXiv Detail & Related papers (2020-05-27T20:32:13Z) - Few-Shot Class-Incremental Learning [68.75462849428196]
We focus on a challenging but practical few-shot class-incremental learning (FSCIL) problem.
FSCIL requires CNN models to incrementally learn new classes from very few labelled samples, without forgetting the previously learned ones.
We represent the knowledge using a neural gas (NG) network, which can learn and preserve the topology of the feature manifold formed by different classes.
arXiv Detail & Related papers (2020-04-23T03:38:33Z) - Aggregated Learning: A Vector-Quantization Approach to Learning Neural
Network Classifiers [48.11796810425477]
We show that IB learning is, in fact, equivalent to a special class of the quantization problem.
We propose a novel learning framework, "Aggregated Learning", for classification with neural network models.
The effectiveness of this framework is verified through extensive experiments on standard image recognition and text classification tasks.
arXiv Detail & Related papers (2020-01-12T16:22:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.