Rethinking Meta-Learning from a Learning Lens

• URL: http://arxiv.org/abs/2409.08474v2
• Date: Mon, 17 Feb 2025 13:22:54 GMT
• Title: Rethinking Meta-Learning from a Learning Lens
• Authors: Jingyao Wang, Wenwen Qiang, Chuxiong Sun, Changwen Zheng, Jiangmeng Li,
• Abstract summary: We focus on the more fundamental learning to learn strategy of meta-learning to explore what causes errors and how to eliminate these errors without changing the environment.<n>We propose using task relations to calibrate the optimization process of meta-learning and propose a plug-and-play method called Task Relation Learner (TRLearner) to achieve this goal.
• Score: 15.36934812747678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Meta-learning has emerged as a powerful approach for leveraging knowledge from previous tasks to solve new tasks. The mainstream methods focus on training a well-generalized model initialization, which is then adapted to different tasks with limited data and updates. However, it pushes the model overfitting on the training tasks. Previous methods mainly attributed this to the lack of data and used augmentations to address this issue, but they were limited by sufficient training and effective augmentation strategies. In this work, we focus on the more fundamental learning to learn strategy of meta-learning to explore what causes errors and how to eliminate these errors without changing the environment. Specifically, we first rethink the algorithmic procedure of meta-learning from a learning lens. Through theoretical and empirical analyses, we find that (i) this paradigm faces the risk of both overfitting and underfitting and (ii) the model adapted to different tasks promote each other where the effect is stronger if the tasks are more similar. Based on this insight, we propose using task relations to calibrate the optimization process of meta-learning and propose a plug-and-play method called Task Relation Learner (TRLearner) to achieve this goal. Specifically, it first obtains task relation matrices from the extracted task-specific meta-data. Then, it uses the obtained matrices with relation-aware consistency regularization to guide optimization. Extensive theoretical and empirical analyses demonstrate the effectiveness of TRLearner.

Related papers

• ConML: A Universal Meta-Learning Framework with Task-Level Contrastive Learning [49.447777286862994]
  ConML is a universal meta-learning framework that can be applied to various meta-learning algorithms. We demonstrate that ConML integrates seamlessly with optimization-based, metric-based, and amortization-based meta-learning algorithms.
  arXiv  Detail & Related papers  (2024-10-08T12:22:10Z)
• Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent [44.56938629818211]
  A fundamental challenge in meta-learning is how to quickly "adapt" the extracted prior in order to train a task-specific model. Existing approaches deal with this challenge using a preconditioner that enhances convergence of the per-task training process. The present contribution addresses this limitation by learning a nonlinear mirror map, which induces a versatile distance metric.
  arXiv  Detail & Related papers  (2023-12-20T23:45:06Z)
• Unsupervised Representation Learning to Aid Semi-Supervised Meta Learning [16.534014215010757]
  We propose a one-shot unsupervised meta-learning to learn latent representation of training samples. A temperature-scaled cross-entropy loss is used in the inner loop of meta-learning to prevent overfitting. The proposed method is model agnostic and can aid any meta-learning model to improve accuracy.
  arXiv  Detail & Related papers  (2023-10-19T18:25:22Z)
• Learning to Learn with Indispensable Connections [6.040904021861969]
  We propose a novel meta-learning method called Meta-LTH that includes indispensible (necessary) connections. Our method improves the classification accuracy by approximately 2% (20-way 1-shot task setting) for omniglot dataset.
  arXiv  Detail & Related papers  (2023-04-06T04:53:13Z)
• On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning [71.55412580325743]
  We show that multi-task pretraining with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation. This is encouraging for future research, as multi-task pretraining tends to be simpler and computationally cheaper than meta-RL.
  arXiv  Detail & Related papers  (2022-06-07T13:24:00Z)
• What Matters For Meta-Learning Vision Regression Tasks? [19.373532562905208]
  This paper makes two main contributions that help understand this barely explored area. First, we design two new types of cross-category level vision regression tasks, namely object discovery and pose estimation. Second, we propose the addition of functional contrastive learning (FCL) over the task representations in Conditional Neural Processes (CNPs) and train in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-03-09T17:28:16Z)
• Modeling and Optimization Trade-off in Meta-learning [23.381986209234164]
  We introduce and rigorously define the trade-off between accurate modeling and ease in meta-learning. Taking MAML as a representative metalearning algorithm, we theoretically characterize the trade-off for general non risk functions as well as linear regression. We also empirically solve a trade-off for metareinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2020-10-24T15:32:08Z)
• Meta-learning the Learning Trends Shared Across Tasks [123.10294801296926]
  Gradient-based meta-learning algorithms excel at quick adaptation to new tasks with limited data. Existing meta-learning approaches only depend on the current task information during the adaptation. We propose a 'Path-aware' model-agnostic meta-learning approach.
  arXiv  Detail & Related papers  (2020-10-19T08:06:47Z)
• Probabilistic Active Meta-Learning [15.432006404678981]
  We introduce task selection based on prior experience into a meta-learning algorithm. We provide empirical evidence that our approach improves data-efficiency when compared to strong baselines on simulated robotic experiments.
  arXiv  Detail & Related papers  (2020-07-17T12:51:42Z)
• Concept Learners for Few-Shot Learning [76.08585517480807]
  We propose COMET, a meta-learning method that improves generalization ability by learning to learn along human-interpretable concept dimensions. We evaluate our model on few-shot tasks from diverse domains, including fine-grained image classification, document categorization and cell type annotation.
  arXiv  Detail & Related papers  (2020-07-14T22:04:17Z)
• Meta-Learning Requires Meta-Augmentation [13.16019567695033]
  We describe two forms of metalearning overfitting, and show that they appear experimentally in common benchmarks. We then use an information-theoretic framework to discuss meta-augmentation, a way to add randomness that discourages the base learner and model from learning trivial solutions. We demonstrate that meta-augmentation produces large complementary benefits to recently proposed meta-regularization techniques.
  arXiv  Detail & Related papers  (2020-07-10T18:04:04Z)
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling [126.69933134648541]
  We present a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time. Our method is based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data.
  arXiv  Detail & Related papers  (2020-06-12T13:34:46Z)
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning [0.0]
  Meta-learning also known as learning-to-learn which seeks rapid and accurate model adaptation to unseen tasks. We briefly introduce meta-learning methodologies in the following categories: black-box meta-learning, metric-based meta-learning, layered meta-learning and Bayesian meta-learning framework.
  arXiv  Detail & Related papers  (2020-04-17T03:11:08Z)
• Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning [79.25478727351604]
  We explore a simple process: meta-learning over a whole-classification pre-trained model on its evaluation metric. We observe this simple method achieves competitive performance to state-of-the-art methods on standard benchmarks.
  arXiv  Detail & Related papers  (2020-03-09T20:06:36Z)
• Meta Cyclical Annealing Schedule: A Simple Approach to Avoiding Meta-Amortization Error [50.83356836818667]
  We develop a novel meta-regularization objective using it cyclical annealing schedule and it maximum mean discrepancy (MMD) criterion. The experimental results show that our approach substantially outperforms standard meta-learning algorithms.
  arXiv  Detail & Related papers  (2020-03-04T04:43:16Z)
• Provable Meta-Learning of Linear Representations [114.656572506859]
  We provide fast, sample-efficient algorithms to address the dual challenges of learning a common set of features from multiple, related tasks, and transferring this knowledge to new, unseen tasks. We also provide information-theoretic lower bounds on the sample complexity of learning these linear features.
  arXiv  Detail & Related papers  (2020-02-26T18:21:34Z)
• Structured Prediction for Conditional Meta-Learning [44.30857707980074]
  We propose a new perspective on conditional meta-learning via structured prediction. We derive task-adaptive structured meta-learning (TASML), a principled framework that yields task-specific objective functions. Empirically, we show that TASML improves the performance of existing meta-learning models, and outperforms the state-of-the-art on benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-20T15:24:15Z)
• Revisiting Meta-Learning as Supervised Learning [69.2067288158133]
  We aim to provide a principled, unifying framework by revisiting and strengthening the connection between meta-learning and traditional supervised learning. By treating pairs of task-specific data sets and target models as (feature, label) samples, we can reduce many meta-learning algorithms to instances of supervised learning. This view not only unifies meta-learning into an intuitive and practical framework but also allows us to transfer insights from supervised learning directly to improve meta-learning.
  arXiv  Detail & Related papers  (2020-02-03T06:13:01Z)
• Automated Relational Meta-learning [95.02216511235191]
  We propose an automated relational meta-learning framework that automatically extracts the cross-task relations and constructs the meta-knowledge graph. We conduct extensive experiments on 2D toy regression and few-shot image classification and the results demonstrate the superiority of ARML over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2020-01-03T07:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.