Meta-Tasks: An alternative view on Meta-Learning Regularization

• URL: http://arxiv.org/abs/2402.18599v1
• Date: Tue, 27 Feb 2024 21:15:40 GMT
• Title: Meta-Tasks: An alternative view on Meta-Learning Regularization
• Authors: Mohammad Rostami, Atik Faysal, Huaxia Wang, Avimanyu Sahoo and Ryan Antle
• Abstract summary: This paper proposes a novel solution that can generalize to both training and novel tasks while also utilizing unlabeled samples. The experimental results show that the proposed method outperforms prototypical networks by 3.9%.
• Score: 17.738450255829633
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Few-shot learning (FSL) is a challenging machine learning problem due to a scarcity of labeled data. The ability to generalize effectively on both novel and training tasks is a significant barrier to FSL. This paper proposes a novel solution that can generalize to both training and novel tasks while also utilizing unlabeled samples. The method refines the embedding model before updating the outer loop using unsupervised techniques as ``meta-tasks''. The experimental results show that our proposed method performs well on novel and training tasks, with faster and better convergence, lower generalization, and standard deviation error, indicating its potential for practical applications in FSL. The experimental results show that the proposed method outperforms prototypical networks by 3.9%.

Related papers

• Model-Based Transfer Learning for Contextual Reinforcement Learning [5.5597941107270215]
  We show how to systematically select good tasks to train, maximizing overall performance across a range of tasks. Key idea behind our approach is to explicitly model the performance loss incurred by transferring a trained model. We experimentally validate our methods using urban traffic and standard control benchmarks.
  arXiv  Detail & Related papers  (2024-08-08T14:46:01Z)
• Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification [34.37262622415682]
  We propose a new adaptation framework called Data Adaptive Traceback. Specifically, we utilize a zero-shot-based method to extract the most downstream task-related subset of the pre-training data. We adopt a pseudo-label-based semi-supervised technique to reuse the pre-training images and a vision-language contrastive learning method to address the confirmation bias issue in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-07-11T18:01:58Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Invariant Meta Learning for Out-of-Distribution Generalization [1.1718589131017048]
  In this paper, we propose invariant meta learning for out-of-distribution tasks. Specifically, invariant optimal meta-initialization,and fast adapt to out-of-distribution tasks with regularization penalty.
  arXiv  Detail & Related papers  (2023-01-26T12:53:21Z)
• Improving Multi-task Learning via Seeking Task-based Flat Regions [43.85516379095757]
  Multi-Task Learning (MTL) is a powerful learning paradigm for training deep neural networks that allows learning more than one objective by a single backbone. There is an emerging line of work in MTL that focuses on manipulating the task gradient to derive an ultimate gradient descent direction. We propose to leverage a recently introduced training method, named Sharpness-aware Minimization, which can enhance model generalization ability on single-task learning.
  arXiv  Detail & Related papers  (2022-11-24T17:19:30Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• A Strong Baseline for Semi-Supervised Incremental Few-Shot Learning [54.617688468341704]
  Few-shot learning aims to learn models that generalize to novel classes with limited training samples. We propose a novel paradigm containing two parts: (1) a well-designed meta-training algorithm for mitigating ambiguity between base and novel classes caused by unreliable pseudo labels and (2) a model adaptation mechanism to learn discriminative features for novel classes while preserving base knowledge using few labeled and all the unlabeled data.
  arXiv  Detail & Related papers  (2021-10-21T13:25:52Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Sample-based Regularization: A Transfer Learning Strategy Toward Better Generalization [8.432864879027724]
  Training a deep neural network with a small amount of data is a challenging problem. One of the practical difficulties that we often face is to collect many samples. By using the source model trained with a large-scale dataset, the target model can alleviate the overfitting originated from the lack of training data.
  arXiv  Detail & Related papers  (2020-07-10T06:02:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.