A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition

• URL: http://arxiv.org/abs/2305.12485v2
• Date: Thu, 27 Jul 2023 10:06:49 GMT
• Title: A Confidence-based Partial Label Learning Model for Crowd-Annotated Named Entity Recognition
• Authors: Limao Xiong, Jie Zhou, Qunxi Zhu, Xiao Wang, Yuanbin Wu, Qi Zhang, Tao Gui, Xuanjing Huang, Jin Ma, Ying Shan
• Abstract summary: Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets. We propose a Confidence-based Partial Label Learning (CPLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER.
• Score: 74.79785063365289
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing models for named entity recognition (NER) are mainly based on large-scale labeled datasets, which always obtain using crowdsourcing. However, it is hard to obtain a unified and correct label via majority voting from multiple annotators for NER due to the large labeling space and complexity of this task. To address this problem, we aim to utilize the original multi-annotator labels directly. Particularly, we propose a Confidence-based Partial Label Learning (CPLL) method to integrate the prior confidence (given by annotators) and posterior confidences (learned by models) for crowd-annotated NER. This model learns a token- and content-dependent confidence via an Expectation-Maximization (EM) algorithm by minimizing empirical risk. The true posterior estimator and confidence estimator perform iteratively to update the true posterior and confidence respectively. We conduct extensive experimental results on both real-world and synthetic datasets, which show that our model can improve performance effectively compared with strong baselines.

Related papers

• Dynamic Correlation Learning and Regularization for Multi-Label Confidence Calibration [60.95748658638956]
  This paper introduces the Multi-Label Confidence task, aiming to provide well-calibrated confidence scores in multi-label scenarios. Existing single-label calibration methods fail to account for category correlations, which are crucial for addressing semantic confusion. We propose the Dynamic Correlation Learning and Regularization algorithm, which leverages multi-grained semantic correlations to better model semantic confusion.
  arXiv  Detail & Related papers  (2024-07-09T13:26:21Z)
• Uncertainty-aware self-training with expectation maximization basis transformation [9.7527450662978]
  We propose a new self-training framework to combine uncertainty information of both model and dataset. Specifically, we propose to use Expectation-Maximization (EM) to smooth the labels and comprehensively estimate the uncertainty information.
  arXiv  Detail & Related papers  (2024-05-02T11:01:31Z)
• CAST: Cluster-Aware Self-Training for Tabular Data via Reliable Confidence [0.4999814847776098]
  Self-training is vulnerable to noisy pseudo-labels caused by erroneous confidence. Cluster-Aware Self-Training (CAST) enhances existing self-training algorithms at a negligible cost.
  arXiv  Detail & Related papers  (2023-10-10T07:46:54Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [50.62245481416744]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• Confidence-Aware Calibration and Scoring Functions for Curriculum Learning [1.192436948211501]
  We integrate notions of model confidence and human confidence with label smoothing to achieve better model calibration and generalization. A higher model or human confidence score indicates a more recognisable and therefore easier sample, and can therefore be used as a scoring function to rank samples in curriculum learning.
  arXiv  Detail & Related papers  (2023-01-29T23:59:40Z)
• How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling? [11.481435098152893]
  This paper aims to provide some empirical insights on estimating model confidence for generative sequence labeling. As verified over six public datasets, we show that our proposed approach significantly reduces calibration errors of the predictions of a generative sequence labeling model.
  arXiv  Detail & Related papers  (2022-12-21T05:01:01Z)
• Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training [66.80558875393565]
  We study the problem of training named entity recognition (NER) models using only distantly-labeled data. We propose a noise-robust learning scheme comprised of a new loss function and a noisy label removal step. Our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.
  arXiv  Detail & Related papers  (2021-09-10T17:19:56Z)
• Learning from Similarity-Confidence Data [94.94650350944377]
  We investigate a novel weakly supervised learning problem of learning from similarity-confidence (Sconf) data. We propose an unbiased estimator of the classification risk that can be calculated from only Sconf data and show that the estimation error bound achieves the optimal convergence rate.
  arXiv  Detail & Related papers  (2021-02-13T07:31:16Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.