When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications

• URL: http://arxiv.org/abs/2005.11700v2
• Date: Wed, 24 Apr 2024 05:06:27 GMT
• Title: When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications
• Authors: Zequn Liu, Ruiyi Zhang, Yiping Song, Wei Ju, Ming Zhang,
• Abstract summary: Many impacting factors, including data quantity, similarity among tasks, and the balance between general language model and task-specific adaptation, can affect the performance of MAML in NLP. In this paper, we conduct an empirical study to investigate these impacting factors and conclude when MAML works the best based on the experimental results.
• Score: 26.458825286934857
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-Agnostic Meta-Learning (MAML), a model-agnostic meta-learning method, is successfully employed in NLP applications including few-shot text classification and multi-domain low-resource language generation. Many impacting factors, including data quantity, similarity among tasks, and the balance between general language model and task-specific adaptation, can affect the performance of MAML in NLP, but few works have thoroughly studied them. In this paper, we conduct an empirical study to investigate these impacting factors and conclude when MAML works the best based on the experimental results.

Related papers

• Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning [15.919493497867567]
  This study aims to evaluate the performance of Multimodal Large Language Models (MLLMs) on the VALSE benchmark. We conducted a comprehensive assessment of state-of-the-art MLLMs, varying in model size and pretraining datasets.
  arXiv  Detail & Related papers  (2024-07-17T11:26:47Z)
• MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning [43.512739869120125]
  We propose MAML-en-LLM, a novel method for meta-training large language models (LLMs) MAML-en-LLM can learn truly generalizable parameters that not only perform well on disjointed tasks but also adapts to unseen tasks. We demonstrate that MAML-en-LLM outperforms baselines in settings with limited amount of training data on both seen and unseen domains.
  arXiv  Detail & Related papers  (2024-05-19T04:49:42Z)
• Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach [64.42462708687921]
  Evaluations have revealed that factors such as scaling, training types, architectures and other factors profoundly impact the performance of LLMs. Our study embarks on a thorough re-examination of these LLMs, targeting the inadequacies in current evaluation methods. This includes the application of ANOVA, Tukey HSD tests, GAMM, and clustering technique.
  arXiv  Detail & Related papers  (2024-03-22T14:47:35Z)
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation [94.59630161324013]
  We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
  arXiv  Detail & Related papers  (2023-10-24T08:56:49Z)
• MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks [56.60050181186531]
  We introduce MM-BigBench, which incorporates a diverse range of metrics to offer an extensive evaluation of the performance of various models and instructions. Our paper evaluates a total of 20 language models (14 MLLMs) on 14 multimodal datasets spanning 6 tasks, with 10 instructions for each task, and derives novel insights.
  arXiv  Detail & Related papers  (2023-10-13T11:57:04Z)
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark [53.663757126289795]
  This paper introduces GAOKAO-Bench, an intuitive benchmark that employs questions from the Chinese GAOKAO examination as test samples. With human evaluation, we obtain the converted total score of LLMs, including GPT-4, ChatGPT and ERNIE-Bot. We also use LLMs to grade the subjective questions, and find that model scores achieve a moderate level of consistency with human scores.
  arXiv  Detail & Related papers  (2023-05-21T14:39:28Z)
• Model-Agnostic Meta-Learning for Natural Language Understanding Tasks in Finance [1.863067234952186]
  We investigate model-agnostic meta-learning algorithm(MAML) in low-resource financial NLU tasks. Our models achieve the state-of-the-art performance according to the experimental results.
  arXiv  Detail & Related papers  (2023-03-06T02:24:48Z)
• ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models [78.08792285698853]
  We present a large-scale empirical study on general language ability evaluation of pretrained language models (ElitePLM) Our empirical results demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; and (3) PLMs have excellent transferability between similar tasks.
  arXiv  Detail & Related papers  (2022-05-03T14:18:10Z)
• Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects [23.347535672670688]
  Multi-Agent Reinforcement Learning (MARL) tackles sequential decision-making problems involving multiple participants. MARL requires a tremendous number of samples for effective training. Model-based methods have been shown to achieve provable advantages of sample efficiency.
  arXiv  Detail & Related papers  (2022-03-20T17:24:47Z)
• Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably? [25.00480072097939]
  We compare the meta test risks of model agnostic meta learning (MAML) and Bayesian MAML. Under both the distribution agnostic and linear centroid cases, we have established that Bayesian MAML indeed has provably lower meta test risks than MAML.
  arXiv  Detail & Related papers  (2022-03-06T21:38:18Z)
• MAML is a Noisy Contrastive Learner [72.04430033118426]
  Model-agnostic meta-learning (MAML) is one of the most popular and widely-adopted meta-learning algorithms nowadays. We provide a new perspective to the working mechanism of MAML and discover that: MAML is analogous to a meta-learner using a supervised contrastive objective function. We propose a simple but effective technique, zeroing trick, to alleviate such interference.
  arXiv  Detail & Related papers  (2021-06-29T12:52:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.