BOIL: Towards Representation Change for Few-shot Learning

• URL: http://arxiv.org/abs/2008.08882v2
• Date: Wed, 3 Mar 2021 05:16:52 GMT
• Title: BOIL: Towards Representation Change for Few-shot Learning
• Authors: Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, Se-Young Yun
• Abstract summary: We study the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks. We propose a novel meta-learning algorithm, called BOIL, which updates only the body of the model and freezes the head during inner loop updates. BOIL empirically shows significant performance improvement over MAML, particularly on cross-domain tasks.
• Score: 20.23766569940024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms. MAML learns new tasks with a few data samples using inner updates from a meta-initialization point and learns the meta-initialization parameters with outer updates. It has recently been hypothesized that representation reuse, which makes little change in efficient representations, is the dominant factor in the performance of the meta-initialized model through MAML in contrast to representation change, which causes a significant change in representations. In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks. To this aim, we propose a novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop), which updates only the body (extractor) of the model and freezes the head (classifier) during inner loop updates. BOIL leverages representation change rather than representation reuse. This is because feature vectors (representations) have to move quickly to their corresponding frozen head vectors. We visualize this property using cosine similarity, CKA, and empirical results without the head. BOIL empirically shows significant performance improvement over MAML, particularly on cross-domain tasks. The results imply that representation change in gradient-based meta-learning approaches is a critical component.

Related papers

• Cooperative Meta-Learning with Gradient Augmentation [6.472397166280681]
  We propose a novel cooperative meta-learning framework dubbed CML. In this work, we propose a novel cooperative meta-learning framework dubbed CML.
  arXiv  Detail & Related papers  (2024-06-07T04:54:00Z)
• Context-Aware Meta-Learning [52.09326317432577]
  We propose a meta-learning algorithm that emulates Large Language Models by learning new visual concepts during inference without fine-tuning. Our approach exceeds or matches the state-of-the-art algorithm, P>M>F, on 8 out of 11 meta-learning benchmarks.
  arXiv  Detail & Related papers  (2023-10-17T03:35:27Z)
• Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation [102.25240608024063]
  Referring image segments an image from a language expression. We develop an algorithm that shifts from being localization-centric to segmentation-language. Compared to its counterparts, our method is more versatile yet effective.
  arXiv  Detail & Related papers  (2023-03-11T08:42:40Z)
• MAC: A Meta-Learning Approach for Feature Learning and Recombination [0.0]
  In this paper, we invoke the width-depth duality of neural networks, wherein, we increase the width of the network by adding extra computational units (ACU) Experimental results show that our proposed MAC method outperformed existing ANIL algorithm for non-similar task distribution by approximately 13%.
  arXiv  Detail & Related papers  (2022-09-20T11:09:54Z)
• Does MAML Only Work via Feature Re-use? A Data Centric Perspective [19.556093984142418]
  We provide empirical results that shed some light on how meta-learned MAML representations function. We show that it is possible to define a family of synthetic benchmarks that result in a low degree of feature re-use. We conjecture the core challenge of re-thinking meta-learning is in the design of few-shot learning data sets and benchmarks.
  arXiv  Detail & Related papers  (2021-12-24T20:18:38Z)
• How to Train Your MAML to Excel in Few-Shot Classification [26.51244463209443]
  We show how to train MAML to excel in few-shot classification. Our approach, which we name UNICORN-MAML, performs on a par with or even outperforms state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2021-06-30T17:56:15Z)
• Exploring the Similarity of Representations in Model-Agnostic Meta-Learning [0.0]
  Model-agnostic meta-learning (MAML) has been one of the most promising approaches in meta-learning. Recent work proposes that MAML rather reuses features than rapidly learns. We apply representation similarity analysis (RSA), a well-established method in neuroscience, to the few-shot learning instantiation of MAML.
  arXiv  Detail & Related papers  (2021-05-12T16:20:40Z)
• How Fine-Tuning Allows for Effective Meta-Learning [50.17896588738377]
  We present a theoretical framework for analyzing representations derived from a MAML-like algorithm. We provide risk bounds on the best predictor found by fine-tuning via gradient descent, demonstrating that the algorithm can provably leverage the shared structure. This separation result underscores the benefit of fine-tuning-based methods, such as MAML, over methods with "frozen representation" objectives in few-shot learning.
  arXiv  Detail & Related papers  (2021-05-05T17:56:00Z)
• Fast Few-Shot Classification by Few-Iteration Meta-Learning [173.32497326674775]
  We introduce a fast optimization-based meta-learning method for few-shot classification. Our strategy enables important aspects of the base learner objective to be learned during meta-training. We perform a comprehensive experimental analysis, demonstrating the speed and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-01T15:59:31Z)
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? [72.00712736992618]
  We show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, outperforms state-of-the-art few-shot learning methods. An additional boost can be achieved through the use of self-distillation. We believe that our findings motivate a rethinking of few-shot image classification benchmarks and the associated role of meta-learning algorithms.
  arXiv  Detail & Related papers  (2020-03-25T17:58:42Z)
• Plannable Approximations to MDP Homomorphisms: Equivariance under Actions [72.30921397899684]
  We introduce a contrastive loss function that enforces action equivariance on the learned representations. We prove that when our loss is zero, we have a homomorphism of a deterministic Markov Decision Process. We show experimentally that for deterministic MDPs, the optimal policy in the abstract MDP can be successfully lifted to the original MDP.
  arXiv  Detail & Related papers  (2020-02-27T08:29:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.