Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim
Verification with Pattern Exploiting Training
- URL: http://arxiv.org/abs/2208.08749v1
- Date: Thu, 18 Aug 2022 10:11:36 GMT
- Title: Active PETs: Active Data Annotation Prioritisation for Few-Shot Claim
Verification with Pattern Exploiting Training
- Authors: Xia Zeng, Arkaitz Zubiaga
- Abstract summary: Active PETs is a weighted approach that actively selects unlabelled data as candidates for annotation.
Using Active PETs for data selection shows consistent improvement over the state-of-the-art active learning method.
Our approach enables effective selection of instances to be labelled where unlabelled data is abundant.
- Score: 21.842139093124512
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: To mitigate the impact of data scarcity on fact-checking systems, we focus on
few-shot claim verification. Despite recent work on few-shot classification by
proposing advanced language models, there is a dearth of research in data
annotation prioritisation that improves the selection of the few shots to be
labelled for optimal model performance. We propose Active PETs, a novel
weighted approach that utilises an ensemble of Pattern Exploiting Training
(PET) models based on various language models, to actively select unlabelled
data as candidates for annotation. Using Active PETs for data selection shows
consistent improvement over the state-of-the-art active learning method, on two
technical fact-checking datasets and using six different pretrained language
models. We show further improvement with Active PETs-o, which further
integrates an oversampling strategy. Our approach enables effective selection
of instances to be labelled where unlabelled data is abundant but resources for
labelling are limited, leading to consistently improved few-shot claim
verification performance. Our code will be available upon publication.
Related papers
- Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines [83.65380507372483]
Large pre-trained models can dramatically reduce the amount of task-specific data required to solve a problem, but they often fail to capture domain-specific nuances out of the box.
This paper shows how to leverage recent advances in NLP and multi-modal learning to augment a pre-trained model with search engine retrieval.
arXiv Detail & Related papers (2023-11-29T05:33:28Z) - ASPEST: Bridging the Gap Between Active Learning and Selective
Prediction [56.001808843574395]
Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain.
Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples.
In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
arXiv Detail & Related papers (2023-04-07T23:51:07Z) - Temporal Output Discrepancy for Loss Estimation-based Active Learning [65.93767110342502]
We present a novel deep active learning approach that queries the oracle for data annotation when the unlabeled sample is believed to incorporate high loss.
Our approach achieves superior performances than the state-of-the-art active learning methods on image classification and semantic segmentation tasks.
arXiv Detail & Related papers (2022-12-20T19:29:37Z) - ALLWAS: Active Learning on Language models in WASserstein space [13.35098213857704]
In several domains, such as medicine, the scarcity of labeled training data is a common issue.
Active learning may prove helpful in these cases to boost the performance with a limited label budget.
We propose a novel method using sampling techniques based on submodular optimization and optimal transport for active learning in language models.
arXiv Detail & Related papers (2021-09-03T18:11:07Z) - Bayesian Active Learning with Pretrained Language Models [9.161353418331245]
Active Learning (AL) is a method to iteratively select data for annotation from a pool of unlabeled data.
Previous AL approaches have been limited to task-specific models that are trained from scratch at each iteration.
We introduce BALM; Bayesian Active Learning with pretrained language models.
arXiv Detail & Related papers (2021-04-16T19:07:31Z) - Just Label What You Need: Fine-Grained Active Selection for Perception
and Prediction through Partially Labeled Scenes [78.23907801786827]
We introduce generalizations that ensure that our approach is both cost-aware and allows for fine-grained selection of examples through partially labeled scenes.
Our experiments on a real-world, large-scale self-driving dataset suggest that fine-grained selection can improve the performance across perception, prediction, and downstream planning tasks.
arXiv Detail & Related papers (2021-04-08T17:57:41Z) - Improving and Simplifying Pattern Exploiting Training [81.77863825517511]
Pattern Exploiting Training (PET) is a recent approach that leverages patterns for few-shot learning.
In this paper, we focus on few shot learning without any unlabeled data and introduce ADAPET.
ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data.
arXiv Detail & Related papers (2021-03-22T15:52:45Z) - Active Testing: Sample-Efficient Model Evaluation [39.200332879659456]
We introduce active testing: a new framework for sample-efficient model evaluation.
Active testing addresses this by carefully selecting the test points to label.
We show how to remove that bias while reducing the variance of the estimator.
arXiv Detail & Related papers (2021-03-09T10:20:49Z) - Semi-supervised Batch Active Learning via Bilevel Optimization [89.37476066973336]
We formulate our approach as a data summarization problem via bilevel optimization.
We show that our method is highly effective in keyword detection tasks in the regime when only few labeled samples are available.
arXiv Detail & Related papers (2020-10-19T16:53:24Z) - Active and Incremental Learning with Weak Supervision [7.2288756536476635]
In this work, we describe combinations of an incremental learning scheme and methods of active learning.
An object detection task is evaluated in a continuous exploration context on the PASCAL VOC dataset.
We also validate a weakly supervised system based on active and incremental learning in a real-world biodiversity application.
arXiv Detail & Related papers (2020-01-20T13:21:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.