Subspace Adaptation Prior for Few-Shot Learning
- URL: http://arxiv.org/abs/2310.09028v1
- Date: Fri, 13 Oct 2023 11:40:18 GMT
- Title: Subspace Adaptation Prior for Few-Shot Learning
- Authors: Mike Huisman, Aske Plaat, Jan N. van Rijn
- Abstract summary: Subspace Adaptation Prior is a novel gradient-based meta-learning algorithm.
We show that SAP yields superior or competitive performance in few-shot image classification settings.
- Score: 5.2997197698288945
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Gradient-based meta-learning techniques aim to distill useful prior knowledge
from a set of training tasks such that new tasks can be learned more
efficiently with gradient descent. While these methods have achieved successes
in various scenarios, they commonly adapt all parameters of trainable layers
when learning new tasks. This neglects potentially more efficient learning
strategies for a given task distribution and may be susceptible to overfitting,
especially in few-shot learning where tasks must be learned from a limited
number of examples. To address these issues, we propose Subspace Adaptation
Prior (SAP), a novel gradient-based meta-learning algorithm that jointly learns
good initialization parameters (prior knowledge) and layer-wise parameter
subspaces in the form of operation subsets that should be adaptable. In this
way, SAP can learn which operation subsets to adjust with gradient descent
based on the underlying task distribution, simultaneously decreasing the risk
of overfitting when learning new tasks. We demonstrate that this ability is
helpful as SAP yields superior or competitive performance in few-shot image
classification settings (gains between 0.1% and 3.9% in accuracy). Analysis of
the learned subspaces demonstrates that low-dimensional operations often yield
high activation strengths, indicating that they may be important for achieving
good few-shot learning performance. For reproducibility purposes, we publish
all our research code publicly.
Related papers
- Meta-Learning Neural Procedural Biases [9.876317838854018]
We propose Neural Procedural Bias Meta-Learning, a novel framework designed to meta-learn task procedural biases.
We show that by meta-learning the procedural biases of a neural network, we can induce strong inductive biases towards a distribution of learning tasks, enabling robust learning performance across many well-established few-shot learning benchmarks.
arXiv Detail & Related papers (2024-06-12T08:09:29Z) - Clustering-based Domain-Incremental Learning [4.835091081509403]
Key challenge in continual learning is the so-called "catastrophic forgetting problem"
We propose an online clustering-based approach on a dynamically updated finite pool of samples or gradients.
We demonstrate the effectiveness of the proposed strategy and its promising performance compared to state-of-the-art methods.
arXiv Detail & Related papers (2023-09-21T13:49:05Z) - Learning to Learn with Indispensable Connections [6.040904021861969]
We propose a novel meta-learning method called Meta-LTH that includes indispensible (necessary) connections.
Our method improves the classification accuracy by approximately 2% (20-way 1-shot task setting) for omniglot dataset.
arXiv Detail & Related papers (2023-04-06T04:53:13Z) - Reinforcement Learning with Success Induced Task Prioritization [68.8204255655161]
We introduce Success Induced Task Prioritization (SITP), a framework for automatic curriculum learning.
The algorithm selects the order of tasks that provide the fastest learning for agents.
We demonstrate that SITP matches or surpasses the results of other curriculum design methods.
arXiv Detail & Related papers (2022-12-30T12:32:43Z) - Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers.
Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters.
We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
arXiv Detail & Related papers (2022-03-30T23:16:07Z) - Meta Navigator: Search for a Good Adaptation Policy for Few-shot
Learning [113.05118113697111]
Few-shot learning aims to adapt knowledge learned from previous tasks to novel tasks with only a limited amount of labeled data.
Research literature on few-shot learning exhibits great diversity, while different algorithms often excel at different few-shot learning scenarios.
We present Meta Navigator, a framework that attempts to solve the limitation in few-shot learning by seeking a higher-level strategy.
arXiv Detail & Related papers (2021-09-13T07:20:01Z) - TAG: Task-based Accumulated Gradients for Lifelong learning [21.779858050277475]
We propose a task-aware system that adapts the learning rate based on the relatedness among tasks.
We empirically show that our proposed adaptive learning rate not only accounts for catastrophic forgetting but also allows positive backward transfer.
arXiv Detail & Related papers (2021-05-11T16:10:32Z) - Rectification-based Knowledge Retention for Continual Learning [49.1447478254131]
Deep learning models suffer from catastrophic forgetting when trained in an incremental learning setting.
We propose a novel approach to address the task incremental learning problem, which involves training a model on new tasks that arrive in an incremental manner.
Our approach can be used in both the zero-shot and non zero-shot task incremental learning settings.
arXiv Detail & Related papers (2021-03-30T18:11:30Z) - Efficient Feature Transformations for Discriminative and Generative
Continual Learning [98.10425163678082]
We propose a simple task-specific feature map transformation strategy for continual learning.
Theses provide powerful flexibility for learning new tasks, achieved with minimal parameters added to the base architecture.
We demonstrate the efficacy and efficiency of our method with an extensive set of experiments in discriminative (CIFAR-100 and ImageNet-1K) and generative sequences of tasks.
arXiv Detail & Related papers (2021-03-25T01:48:14Z) - Gradient Projection Memory for Continual Learning [5.43185002439223]
The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems.
We propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks.
arXiv Detail & Related papers (2021-03-17T16:31:29Z) - Meta-learning the Learning Trends Shared Across Tasks [123.10294801296926]
Gradient-based meta-learning algorithms excel at quick adaptation to new tasks with limited data.
Existing meta-learning approaches only depend on the current task information during the adaptation.
We propose a 'Path-aware' model-agnostic meta-learning approach.
arXiv Detail & Related papers (2020-10-19T08:06:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.