Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach

• URL: http://arxiv.org/abs/2402.14948v3
• Date: Mon, 16 Dec 2024 22:39:47 GMT
• Title: Re-Examine Distantly Supervised NER: A New Benchmark and a Simple Approach
• Authors: Yuepei Li, Kang Zhou, Qiao Qiao, Qing Wang, Qi Li,
• Abstract summary: We introduce a real-life DS-NER dataset, QTL, where the training data is annotated using domain dictionaries and the test data is annotated by domain experts.<n>Existing DS-NER approaches fail when applied to QTL, which motivate us to re-examine existing DS-NER approaches.<n>We propose a new approach, token-level Curriculum-based Positive-Unlabeled Learning (CuPUL), which uses curriculum learning to order training samples from easy to hard.
• Score: 14.801411392475439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distantly-Supervised Named Entity Recognition (DS-NER) uses knowledge bases or dictionaries for annotations, reducing manual efforts but rely on large human labeled validation set. In this paper, we introduce a real-life DS-NER dataset, QTL, where the training data is annotated using domain dictionaries and the test data is annotated by domain experts. This dataset has a small validation set, reflecting real-life scenarios. Existing DS-NER approaches fail when applied to QTL, which motivate us to re-examine existing DS-NER approaches. We found that many of them rely on large validation sets and some used test set for tuning inappropriately. To solve this issue, we proposed a new approach, token-level Curriculum-based Positive-Unlabeled Learning (CuPUL), which uses curriculum learning to order training samples from easy to hard. This method stabilizes training, making it robust and effective on small validation sets. CuPUL also addresses false negative issues using the Positive-Unlabeled learning paradigm, demonstrating improved performance in real-life applications.

Related papers

• DynClean: Training Dynamics-based Label Cleaning for Distantly-Supervised Named Entity Recognition [49.54155332262579]
  We propose a training dynamics-based label cleaning approach, which leverages the behavior of a model as training progresses. We also introduce an automatic threshold estimation strategy to locate the errors in distant labels. Our method outperforms numerous advanced DS-NER approaches across four datasets.
  arXiv  Detail & Related papers  (2025-04-06T20:54:42Z)
• LLM Critics Help Catch Bugs in Mathematics: Towards a Better Mathematical Verifier with Natural Language Feedback [71.95402654982095]
  We propose Math-Minos, a natural language feedback-enhanced verifier. Our experiments reveal that a small set of natural language feedback can significantly boost the performance of the verifier.
  arXiv  Detail & Related papers  (2024-06-20T06:42:27Z)
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection [82.43311784594384]
  Real-world datasets contain not only noisy labels but also class imbalance. We propose a simple yet effective method to address noisy labels in imbalanced datasets.
  arXiv  Detail & Related papers  (2024-02-17T10:34:53Z)
• Improving a Named Entity Recognizer Trained on Noisy Data with a Few Clean Instances [55.37242480995541]
  We propose to denoise noisy NER data with guidance from a small set of clean instances. Along with the main NER model we train a discriminator model and use its outputs to recalibrate the sample weights. Results on public crowdsourcing and distant supervision datasets show that the proposed method can consistently improve performance with a small guidance set.
  arXiv  Detail & Related papers  (2023-10-25T17:23:37Z)
• Revisiting Class Imbalance for End-to-end Semi-Supervised Object Detection [1.6249267147413524]
  Semi-supervised object detection (SSOD) has made significant progress with the development of pseudo-label-based end-to-end methods. Many methods face challenges due to class imbalance, which hinders the effectiveness of the pseudo-label generator. In this paper, we examine the root causes of low-quality pseudo-labels and present novel learning mechanisms to improve the label generation quality.
  arXiv  Detail & Related papers  (2023-06-04T06:01:53Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• Label Noise-Robust Learning using a Confidence-Based Sieving Strategy [15.997774467236352]
  In learning tasks with label noise, improving model robustness against overfitting is a pivotal challenge. Identifying the samples with noisy labels and preventing the model from learning them is a promising approach to address this challenge. We propose a novel discriminator metric called confidence error and a sieving strategy called CONFES to differentiate between the clean and noisy samples effectively.
  arXiv  Detail & Related papers  (2022-10-11T10:47:28Z)
• An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning [58.59343434538218]
  We propose a simple but quite effective approach to predict accurate negative pseudo-labels of unlabeled data from an indirect learning perspective. Our approach can be implemented in just few lines of code by only using off-the-shelf operations.
  arXiv  Detail & Related papers  (2022-09-28T02:11:34Z)
• Meta Objective Guided Disambiguation for Partial Label Learning [44.05801303440139]
  We propose a novel framework for partial label learning with meta objective guided disambiguation (MoGD) MoGD aims to recover the ground-truth label from candidate labels set by solving a meta objective on a small validation set. The proposed method can be easily implemented by using various deep networks with the ordinary SGD.
  arXiv  Detail & Related papers  (2022-08-26T06:48:01Z)
• Towards Harnessing Feature Embedding for Robust Learning with Noisy Labels [44.133307197696446]
  The memorization effect of deep neural networks (DNNs) plays a pivotal role in recent label noise learning methods. We propose a novel feature embedding-based method for deep learning with label noise, termed LabEl NoiseDilution (LEND)
  arXiv  Detail & Related papers  (2022-06-27T02:45:09Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Uncertainty-aware Pseudo-label Selection for Positive-Unlabeled Learning [10.014356492742074]
  We propose to tackle the issues of imbalanced datasets and model calibration in a positive-unlabeled learning setting. By boosting the signal from the minority class, pseudo-labeling expands the labeled dataset with new samples from the unlabeled set. Within a series of experiments, PUUPL yields substantial performance gains in highly imbalanced settings.
  arXiv  Detail & Related papers  (2022-01-31T12:55:47Z)
• S3: Supervised Self-supervised Learning under Label Noise [53.02249460567745]
  In this paper we address the problem of classification in the presence of label noise. In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space. Our method significantly surpasses previous methods on both CIFARCIFAR100 with artificial noise and real-world noisy datasets such as WebVision and ANIMAL-10N.
  arXiv  Detail & Related papers  (2021-11-22T15:49:20Z)
• Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training [66.80558875393565]
  We study the problem of training named entity recognition (NER) models using only distantly-labeled data. We propose a noise-robust learning scheme comprised of a new loss function and a noisy label removal step. Our method achieves superior performance, outperforming existing distantly-supervised NER models by significant margins.
  arXiv  Detail & Related papers  (2021-09-10T17:19:56Z)
• Few-shot Learning via Dependency Maximization and Instance Discriminant Analysis [21.8311401851523]
  We study the few-shot learning problem, where a model learns to recognize new objects with extremely few labeled data per category. We propose a simple approach to exploit unlabeled data accompanying the few-shot task for improving few-shot performance.
  arXiv  Detail & Related papers  (2021-09-07T02:19:01Z)
• Boosting Semi-Supervised Face Recognition with Noise Robustness [54.342992887966616]
  This paper presents an effective solution to semi-supervised face recognition that is robust to the label noise aroused by the auto-labelling. We develop a semi-supervised face recognition solution, named Noise Robust Learning-Labelling (NRoLL), which is based on the robust training ability empowered by GN.
  arXiv  Detail & Related papers  (2021-05-10T14:43:11Z)
• Data-efficient Weakly-supervised Learning for On-line Object Detection under Domain Shift in Robotics [24.878465999976594]
  Several object detection methods have been proposed in the literature, the vast majority based on Deep Convolutional Neural Networks (DCNNs) These methods have important limitations for robotics: Learning solely on off-line data may introduce biases, and prevents adaptation to novel tasks. In this work, we investigate how weakly-supervised learning can cope with these problems.
  arXiv  Detail & Related papers  (2020-12-28T16:36:11Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)
• Ask-n-Learn: Active Learning via Reliable Gradient Representations for Image Classification [29.43017692274488]
  Deep predictive models rely on human supervision in the form of labeled training data. We propose Ask-n-Learn, an active learning approach based on gradient embeddings obtained using the pesudo-labels estimated in each of the algorithm.
  arXiv  Detail & Related papers  (2020-09-30T05:19:56Z)
• BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision [49.42215511723874]
  We propose a new computational framework -- BOND -- to improve the prediction performance of NER models. Specifically, we propose a two-stage training algorithm: In the first stage, we adapt the pre-trained language model to the NER tasks using the distant labels. In the second stage, we drop the distant labels, and propose a self-training approach to further improve the model performance.
  arXiv  Detail & Related papers  (2020-06-28T04:55:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.