MEAL: Stable and Active Learning for Few-Shot Prompting

• URL: http://arxiv.org/abs/2211.08358v3
• Date: Mon, 20 Nov 2023 14:52:43 GMT
• Title: MEAL: Stable and Active Learning for Few-Shot Prompting
• Authors: Abdullatif K\"oksal, Timo Schick, Hinrich Sch\"utze
• Abstract summary: Few-shot classification has high variance both across different sets of few shots and across different finetuning runs. We propose novel ensembling methods and show that they substantially reduce run variability. Second, we introduce a new active learning (AL) criterion for data selection and present the first AL-based approach specifically tailored towards prompt-based learning.
• Score: 26.60924937965494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high variance both across different sets of few shots (data selection) and across different finetuning runs (run variability). This is problematic not only because it impedes the fair comparison of different approaches, but especially because it makes few-shot learning too unreliable for many real-world applications. To alleviate these issues, we make two contributions for more stable and effective few-shot learning: First, we propose novel ensembling methods and show that they substantially reduce run variability. Second, we introduce a new active learning (AL) criterion for data selection and present the first AL-based approach specifically tailored towards prompt-based learning. In our experiments, we show that our combined method, MEAL (Multiprompt finetuning and prediction Ensembling with Active Learning), improves overall performance of prompt-based finetuning by 2.3 points on five diverse tasks. We publicly share our code and data splits in https://github.com/akoksal/MEAL.

Related papers

• Exploring Effective Knowledge Transfer for Few-shot Object Detection [54.45009859654753]
  Methods that excel in low-shot regimes are likely to struggle in high-shot regimes, and vice versa. In the low-shot regime, the primary challenge is the lack of inner-class variation. In the high-shot regime, as the variance approaches the real one, the main hindrance to the performance comes from misalignment between learned and true distributions.
  arXiv  Detail & Related papers  (2022-10-05T04:53:58Z)
• A Lagrangian Duality Approach to Active Learning [119.36233726867992]
  We consider the batch active learning problem, where only a subset of the training data is labeled. We formulate the learning problem using constrained optimization, where each constraint bounds the performance of the model on labeled samples. We show, via numerical experiments, that our proposed approach performs similarly to or better than state-of-the-art active learning methods.
  arXiv  Detail & Related papers  (2022-02-08T19:18:49Z)
• An Investigation of Replay-based Approaches for Continual Learning [79.0660895390689]
  Continual learning (CL) is a major challenge of machine learning (ML) and describes the ability to learn several tasks sequentially without catastrophic forgetting (CF) Several solution classes have been proposed, of which so-called replay-based approaches seem very promising due to their simplicity and robustness. We empirically investigate replay-based approaches of continual learning and assess their potential for applications.
  arXiv  Detail & Related papers  (2021-08-15T15:05:02Z)
• Few-Shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning [51.03781020616402]
  Fine-grained action recognition is attracting increasing attention due to the emerging demand of specific action understanding in real-world applications. We propose a few-shot fine-grained action recognition problem, aiming to recognize novel fine-grained actions with only few samples given for each class. Although progress has been made in coarse-grained actions, existing few-shot recognition methods encounter two issues handling fine-grained actions.
  arXiv  Detail & Related papers  (2021-08-15T02:21:01Z)
• MetaKernel: Learning Variational Random Features with Limited Labels [120.90737681252594]
  Few-shot learning deals with the fundamental and challenging problem of learning from a few annotated samples, while being able to generalize well on new tasks. We propose meta-learning kernels with random Fourier features for few-shot learning, we call Meta Kernel.
  arXiv  Detail & Related papers  (2021-05-08T21:24:09Z)
• Towards Few-Shot Fact-Checking via Perplexity [40.11397284006867]
  We propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. Our methodology can already outperform the Major Class baseline by more than absolute 10% on the F1-Macro metric. We construct and publicly release two new fact-checking datasets related to COVID-19.
  arXiv  Detail & Related papers  (2021-03-17T09:43:19Z)
• Few-shot Action Recognition with Prototype-centered Attentive Learning [88.10852114988829]
  Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional query-centered learning objective. Second, PAL integrates a attentive hybrid learning mechanism that can minimize the negative impacts of outliers.
  arXiv  Detail & Related papers  (2021-01-20T11:48:12Z)
• A Primal-Dual Subgradient Approachfor Fair Meta Learning [23.65344558042896]
  Few shot meta-learning is well-known with its fast-adapted capability and accuracy generalization onto unseen tasks. We propose a Primal-Dual Fair Meta-learning framework, namely PDFM, which learns to train fair machine learning models using only a few examples.
  arXiv  Detail & Related papers  (2020-09-26T19:47:38Z)
• Online Parameter-Free Learning of Multiple Low Variance Tasks [36.08679456245112]
  We propose a method to learn a common bias vector for a growing sequence of low-variance tasks. Our approach is presented in the non-statistical setting and can be of two variants. Experiments confirm the effectiveness of our methods in practice.
  arXiv  Detail & Related papers  (2020-07-11T09:52:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.