ATM: An Uncertainty-aware Active Self-training Framework for Label-efficient Text Classification

• URL: http://arxiv.org/abs/2112.08787v1
• Date: Thu, 16 Dec 2021 11:09:48 GMT
• Title: ATM: An Uncertainty-aware Active Self-training Framework for Label-efficient Text Classification
• Authors: Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, Chao Zhang
• Abstract summary: ATM is a new framework that leverage self-training to exploit unlabeled data and is agnostic to the specific AL algorithm. We demonstrate that ATM outperforms the strongest active learning and self-training baselines and improve the label efficiency by 51.9% on average.
• Score: 13.881283744970979
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the great success of pre-trained language models (LMs) in many natural language processing (NLP) tasks, they require excessive labeled data for fine-tuning to achieve satisfactory performance. To enhance the label efficiency, researchers have resorted to active learning (AL), while the potential of unlabeled data is ignored by most of prior work. To unleash the power of unlabeled data for better label efficiency and model performance, we develop ATM, a new framework that leverage self-training to exploit unlabeled data and is agnostic to the specific AL algorithm, serving as a plug-in module to improve existing AL methods. Specifically, the unlabeled data with high uncertainty is exposed to oracle for annotations while those with low uncertainty are leveraged for self-training. To alleviate the label noise propagation issue in self-training, we design a simple and effective momentum-based memory bank to dynamically aggregate the model predictions from all rounds. By extensive experiments, we demonstrate that ATM outperforms the strongest active learning and self-training baselines and improve the label efficiency by 51.9% on average.

Related papers

• Incremental Self-training for Semi-supervised Learning [56.57057576885672]
  IST is simple yet effective and fits existing self-training-based semi-supervised learning methods. We verify the proposed IST on five datasets and two types of backbone, effectively improving the recognition accuracy and learning speed.
  arXiv  Detail & Related papers  (2024-04-14T05:02:00Z)
• MyriadAL: Active Few Shot Learning for Histopathology [10.652626309100889]
  We introduce an active few shot learning framework, Myriad Active Learning (MAL) MAL includes a contrastive-learning encoder, pseudo-label generation, and novel query sample selection in the loop. Experiments on two public histopathology datasets show that MAL has superior test accuracy, macro F1-score, and label efficiency compared to prior works.
  arXiv  Detail & Related papers  (2023-10-24T20:08:15Z)
• XAL: EXplainable Active Learning Makes Classifiers Better Low-resource Learners [71.8257151788923]
  We propose a novel Explainable Active Learning framework (XAL) for low-resource text classification. XAL encourages classifiers to justify their inferences and delve into unlabeled data for which they cannot provide reasonable explanations. Experiments on six datasets show that XAL achieves consistent improvement over 9 strong baselines.
  arXiv  Detail & Related papers  (2023-10-09T08:07:04Z)
• Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling [29.744621356187764]
  This paper presents SeqUST, a novel uncertain-aware self-training framework for Neural sequence labeling (NSL) We incorporate Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation at the token level and then select reliable language tokens from unlabeled data. A well-designed masked sequence labeling task with a noise-robust loss supports robust training, which aims to suppress the problem of noisy pseudo labels.
  arXiv  Detail & Related papers  (2023-02-17T02:40:04Z)
• Towards Harnessing Feature Embedding for Robust Learning with Noisy Labels [44.133307197696446]
  The memorization effect of deep neural networks (DNNs) plays a pivotal role in recent label noise learning methods. We propose a novel feature embedding-based method for deep learning with label noise, termed LabEl NoiseDilution (LEND)
  arXiv  Detail & Related papers  (2022-06-27T02:45:09Z)
• Self-Tuning for Data-Efficient Deep Learning [75.34320911480008]
  Self-Tuning is a novel approach to enable data-efficient deep learning. It unifies the exploration of labeled and unlabeled data and the transfer of a pre-trained model. It outperforms its SSL and TL counterparts on five tasks by sharp margins.
  arXiv  Detail & Related papers  (2021-02-25T14:56:19Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)
• Adaptive Self-training for Few-shot Neural Sequence Labeling [55.43109437200101]
  We develop techniques to address the label scarcity challenge for neural sequence labeling models. Self-training serves as an effective mechanism to learn from large amounts of unlabeled data. meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels.
  arXiv  Detail & Related papers  (2020-10-07T22:29:05Z)
• Uncertainty-aware Self-training for Text Classification with Few Labels [54.13279574908808]
  We study self-training as one of the earliest semi-supervised learning approaches to reduce the annotation bottleneck. We propose an approach to improve self-training by incorporating uncertainty estimates of the underlying neural network. We show our methods leveraging only 20-30 labeled samples per class for each task for training and for validation can perform within 3% of fully supervised pre-trained language models.
  arXiv  Detail & Related papers  (2020-06-27T08:13:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.