Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling

• URL: http://arxiv.org/abs/2302.08659v1
• Date: Fri, 17 Feb 2023 02:40:04 GMT
• Title: Uncertainty-aware Self-training for Low-resource Neural Sequence Labeling
• Authors: Jianing Wang, Chengyu Wang, Jun Huang, Ming Gao, Aoying Zhou
• Abstract summary: This paper presents SeqUST, a novel uncertain-aware self-training framework for Neural sequence labeling (NSL) We incorporate Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation at the token level and then select reliable language tokens from unlabeled data. A well-designed masked sequence labeling task with a noise-robust loss supports robust training, which aims to suppress the problem of noisy pseudo labels.
• Score: 29.744621356187764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural sequence labeling (NSL) aims at assigning labels for input language tokens, which covers a broad range of applications, such as named entity recognition (NER) and slot filling, etc. However, the satisfying results achieved by traditional supervised-based approaches heavily depend on the large amounts of human annotation data, which may not be feasible in real-world scenarios due to data privacy and computation efficiency issues. This paper presents SeqUST, a novel uncertain-aware self-training framework for NSL to address the labeled data scarcity issue and to effectively utilize unlabeled data. Specifically, we incorporate Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation at the token level and then select reliable language tokens from unlabeled data based on the model confidence and certainty. A well-designed masked sequence labeling task with a noise-robust loss supports robust training, which aims to suppress the problem of noisy pseudo labels. In addition, we develop a Gaussian-based consistency regularization technique to further improve the model robustness on Gaussian-distributed perturbed representations. This effectively alleviates the over-fitting dilemma originating from pseudo-labeled augmented data. Extensive experiments over six benchmarks demonstrate that our SeqUST framework effectively improves the performance of self-training, and consistently outperforms strong baselines by a large margin in low-resource scenarios

Related papers

• An Embedding is Worth a Thousand Noisy Labels [0.11999555634662634]
  We propose WANN, a weighted Adaptive Nearest Neighbor approach to address label noise. We show WANN outperforms reference methods on diverse datasets of varying size and under various noise types and severities. Our approach, emphasizing efficiency and explainability, emerges as a simple, robust solution to overcome the inherent limitations of deep neural network training.
  arXiv  Detail & Related papers  (2024-08-26T15:32:31Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• ATM: An Uncertainty-aware Active Self-training Framework for Label-efficient Text Classification [13.881283744970979]
  ATM is a new framework that leverage self-training to exploit unlabeled data and is agnostic to the specific AL algorithm. We demonstrate that ATM outperforms the strongest active learning and self-training baselines and improve the label efficiency by 51.9% on average.
  arXiv  Detail & Related papers  (2021-12-16T11:09:48Z)
• Robustness and reliability when training with noisy labels [12.688634089849023]
  Labelling of data for supervised learning can be costly and time-consuming. Deep neural networks have proved capable of fitting random labels, regularisation and the use of robust loss functions.
  arXiv  Detail & Related papers  (2021-10-07T10:30:20Z)
• Boosting Semi-Supervised Face Recognition with Noise Robustness [54.342992887966616]
  This paper presents an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. We develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN.
  arXiv  Detail & Related papers  (2021-05-10T14:43:11Z)
• Self-Tuning for Data-Efficient Deep Learning [75.34320911480008]
  Self-Tuning is a novel approach to enable data-efficient deep learning. It unifies the exploration of labeled and unlabeled data and the transfer of a pre-trained model. It outperforms its SSL and TL counterparts on five tasks by sharp margins.
  arXiv  Detail & Related papers  (2021-02-25T14:56:19Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)
• Tackling Instance-Dependent Label Noise via a Universal Probabilistic Model [80.91927573604438]
  This paper proposes a simple yet universal probabilistic model, which explicitly relates noisy labels to their instances. Experiments on datasets with both synthetic and real-world label noise verify that the proposed method yields significant improvements on robustness.
  arXiv  Detail & Related papers  (2021-01-14T05:43:51Z)
• Temporal Calibrated Regularization for Robust Noisy Label Learning [60.90967240168525]
  Deep neural networks (DNNs) exhibit great success on many tasks with the help of large-scale well annotated datasets. However, labeling large-scale data can be very costly and error-prone so that it is difficult to guarantee the annotation quality. We propose a Temporal Calibrated Regularization (TCR) in which we utilize the original labels and the predictions in the previous epoch together.
  arXiv  Detail & Related papers  (2020-07-01T04:48:49Z)
• NAT: Noise-Aware Training for Robust Neural Sequence Labeling [30.91638109413785]
  We propose two Noise-Aware Training (NAT) objectives that improve robustness of sequence labeling performed on input. Our data augmentation method trains a neural model using a mixture of clean and noisy samples, whereas our stability training algorithm encourages the model to create a noise-invariant latent representation. Experiments on English and German named entity recognition benchmarks confirmed that NAT consistently improved robustness of popular sequence labeling models.
  arXiv  Detail & Related papers  (2020-05-14T17:30:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.