Conditional Meta-Learning of Linear Representations

• URL: http://arxiv.org/abs/2103.16277v1
• Date: Tue, 30 Mar 2021 12:02:14 GMT
• Title: Conditional Meta-Learning of Linear Representations
• Authors: Giulia Denevi, Massimiliano Pontil, Carlo Ciliberto
• Abstract summary: Standard meta-learning for representation learning aims to find a common representation to be shared across multiple tasks. In this work we overcome this issue by inferring a conditioning function, mapping the tasks' side information into a representation tailored to the task at hand. We propose a meta-algorithm capable of leveraging this advantage in practice.
• Score: 57.90025697492041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard meta-learning for representation learning aims to find a common representation to be shared across multiple tasks. The effectiveness of these methods is often limited when the nuances of the tasks' distribution cannot be captured by a single representation. In this work we overcome this issue by inferring a conditioning function, mapping the tasks' side information (such as the tasks' training dataset itself) into a representation tailored to the task at hand. We study environments in which our conditional strategy outperforms standard meta-learning, such as those in which tasks can be organized in separate clusters according to the representation they share. We then propose a meta-algorithm capable of leveraging this advantage in practice. In the unconditional setting, our method yields a new estimator enjoying faster learning rates and requiring less hyper-parameters to tune than current state-of-the-art methods. Our results are supported by preliminary experiments.

Related papers

• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Set-based Meta-Interpolation for Few-Task Meta-Learning [79.4236527774689]
  We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, to densify the meta-training task distribution. We empirically validate the efficacy of Meta-Interpolation on eight datasets spanning across various domains.
  arXiv  Detail & Related papers  (2022-05-20T06:53:03Z)
• A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning [14.720411598827365]
  splitting data from each task into train and validation sets during meta-training. We argue that the train-validation split encourages the learned representation to be low-rank without compromising on expressivity. Since sample efficiency benefits from low-rankness, the splitting strategy will require very few samples to solve unseen test tasks.
  arXiv  Detail & Related papers  (2021-06-29T17:59:33Z)
• How Fine-Tuning Allows for Effective Meta-Learning [50.17896588738377]
  We present a theoretical framework for analyzing representations derived from a MAML-like algorithm. We provide risk bounds on the best predictor found by fine-tuning via gradient descent, demonstrating that the algorithm can provably leverage the shared structure. This separation result underscores the benefit of fine-tuning-based methods, such as MAML, over methods with "frozen representation" objectives in few-shot learning.
  arXiv  Detail & Related papers  (2021-05-05T17:56:00Z)
• Function Contrastive Learning of Transferable Meta-Representations [38.31692245188669]
  We study the implications of joint training on the transferability of the meta-representations. We propose a decoupled encoder-decoder approach to supervised meta-learning.
  arXiv  Detail & Related papers  (2020-10-14T13:50:22Z)
• The Advantage of Conditional Meta-Learning for Biased Regularization and Fine-Tuning [50.21341246243422]
  Biased regularization and fine-tuning are two recent meta-learning approaches. We propose conditional meta-learning, inferring a conditioning function mapping task's side information into a meta- parameter vector. We then propose a convex meta-algorithm providing a comparable advantage also in practice.
  arXiv  Detail & Related papers  (2020-08-25T07:32:16Z)
• Adaptive Task Sampling for Meta-Learning [79.61146834134459]
  Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time. We propose an adaptive task sampling method to improve the generalization performance.
  arXiv  Detail & Related papers  (2020-07-17T03:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.