Single versus Multiple Annotation for Named Entity Recognition of Mutations

• URL: http://arxiv.org/abs/2101.07450v1
• Date: Tue, 19 Jan 2021 03:54:17 GMT
• Title: Single versus Multiple Annotation for Named Entity Recognition of Mutations
• Authors: David Martinez Iraola and Antonio Jimeno Yepes
• Abstract summary: We address the impact of using a single annotator vs two annotators, in order to measure whether multiple annotators are required. Once we evaluate the performance loss when using a single annotator, we apply different methods to sample the training data for second annotation. We use held-out double-annotated data to build two scenarios with different types of rankings: similarity-based and confidence based. We evaluate both approaches on: (i) their ability to identify training instances that are erroneous, and (ii) on Mutation NER performance for state-of-the-art
• Score: 4.213427823201119
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The focus of this paper is to address the knowledge acquisition bottleneck for Named Entity Recognition (NER) of mutations, by analysing different approaches to build manually-annotated data. We address first the impact of using a single annotator vs two annotators, in order to measure whether multiple annotators are required. Once we evaluate the performance loss when using a single annotator, we apply different methods to sample the training data for second annotation, aiming at improving the quality of the dataset without requiring a full pass. We use held-out double-annotated data to build two scenarios with different types of rankings: similarity-based and confidence based. We evaluate both approaches on: (i) their ability to identify training instances that are erroneous (cases where single-annotator labels differ from double-annotation after discussion), and (ii) on Mutation NER performance for state-of-the-art classifiers after integrating the fixes at different thresholds.

Related papers

• Do Smaller Language Models Answer Contextualised Questions Through Memorisation Or Generalisation? [8.51696622847778]
  A distinction is often drawn between a model's ability to predict a label for an evaluation sample that is directly memorised from highly similar training samples. We propose a method of identifying evaluation samples for which it is very unlikely our model would have memorised the answers.
  arXiv  Detail & Related papers  (2023-11-21T04:06:08Z)
• ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation [35.10805667891489]
  Active learning, as an annotation cost-saving strategy, has not been fully explored in the context of learning from disagreement. We show that in the active learning setting, a multi-head model performs significantly better than a single-head model in terms of uncertainty estimation.
  arXiv  Detail & Related papers  (2023-10-23T14:26:43Z)
• Convolutional autoencoder-based multimodal one-class classification [80.52334952912808]
  One-class classification refers to approaches of learning using data from a single class only. We propose a deep learning one-class classification method suitable for multimodal data.
  arXiv  Detail & Related papers  (2023-09-25T12:31:18Z)
• Drawing the Same Bounding Box Twice? Coping Noisy Annotations in Object Detection with Repeated Labels [6.872072177648135]
  We propose a novel localization algorithm that adapts well-established ground truth estimation methods. Our algorithm also shows superior performance during training on the TexBiG dataset.
  arXiv  Detail & Related papers  (2023-09-18T13:08:44Z)
• Multimodal Fusion Interactions: A Study of Human and Automatic Quantification [116.55145773123132]
  We study how humans annotate two categorizations of multimodal interactions. We propose a method to automatically convert annotations of partial and counterfactual labels to information decomposition.
  arXiv  Detail & Related papers  (2023-06-07T03:44:50Z)
• Class-Specific Variational Auto-Encoder for Content-Based Image Retrieval [95.42181254494287]
  We propose a regularized loss for Variational Auto-Encoders (VAEs) forcing the model to focus on a given class of interest. As a result, the model learns to discriminate the data belonging to the class of interest from any other possibility. Experimental results show that the proposed method outperforms its competition in both in-domain and out-of-domain retrieval problems.
  arXiv  Detail & Related papers  (2023-04-23T19:51:25Z)
• Utilizing supervised models to infer consensus labels and their quality from data with multiple annotators [16.79939549201032]
  Real-world data for classification is often labeled by multiple annotators. We introduce CROWDLAB, a straightforward approach to estimate such data. Our proposed method provides superior estimates for (1)- (3) than many alternative algorithms.
  arXiv  Detail & Related papers  (2022-10-13T07:54:07Z)
• Disjoint Contrastive Regression Learning for Multi-Sourced Annotations [10.159313152511919]
  Large-scale datasets are important for the development of deep learning models. Multiple annotators may be employed to label different subsets of the data. The inconsistency and bias among different annotators are harmful to the model training.
  arXiv  Detail & Related papers  (2021-12-31T12:39:04Z)
• Cross-domain Speech Recognition with Unsupervised Character-level Distribution Matching [60.8427677151492]
  We propose CMatch, a Character-level distribution matching method to perform fine-grained adaptation between each character in two domains. Experiments on the Libri-Adapt dataset show that our proposed approach achieves 14.39% and 16.50% relative Word Error Rate (WER) reduction on both cross-device and cross-environment ASR.
  arXiv  Detail & Related papers  (2021-04-15T14:36:54Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Few-shot Learning for Multi-label Intent Detection [59.66787898744991]
  State-of-the-art work estimates label-instance relevance scores and uses a threshold to select multiple associated intent labels. Experiments on two datasets show that the proposed model significantly outperforms strong baselines in both one-shot and five-shot settings.
  arXiv  Detail & Related papers  (2020-10-11T14:42:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.