Pre-trained Language Models as Re-Annotators
- URL: http://arxiv.org/abs/2205.05368v1
- Date: Wed, 11 May 2022 09:28:23 GMT
- Title: Pre-trained Language Models as Re-Annotators
- Authors: Chang Shu
- Abstract summary: We investigate how to acquire semantic sensitive annotation representations from Pre-trained Language Models.
We fine-tune the Pre-trained Language Models based with cross-validation for annotation correction.
We study the re-annotation in relation extraction and create a new manually revised dataset, Re-DocRED.
- Score: 3.193067591317475
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Annotation noise is widespread in datasets, but manually revising a flawed
corpus is time-consuming and error-prone. Hence, given the prior knowledge in
Pre-trained Language Models and the expected uniformity across all annotations,
we attempt to reduce annotation noise in the corpus through two tasks
automatically: (1) Annotation Inconsistency Detection that indicates the
credibility of annotations, and (2) Annotation Error Correction that rectifies
the abnormal annotations.
We investigate how to acquire semantic sensitive annotation representations
from Pre-trained Language Models, expecting to embed the examples with
identical annotations to the mutually adjacent positions even without
fine-tuning. We proposed a novel credibility score to reveal the likelihood of
annotation inconsistencies based on the neighbouring consistency. Then, we
fine-tune the Pre-trained Language Models based classifier with
cross-validation for annotation correction. The annotation corrector is further
elaborated with two approaches: (1) soft labelling by Kernel Density Estimation
and (2) a novel distant-peer contrastive loss.
We study the re-annotation in relation extraction and create a new manually
revised dataset, Re-DocRED, for evaluating document-level re-annotation. The
proposed credibility scores show promising agreement with human revisions,
achieving a Binary F1 of 93.4 and 72.5 in detecting inconsistencies on TACRED
and DocRED respectively. Moreover, the neighbour-aware classifiers based on
distant-peer contrastive learning and uncertain labels achieve Macro F1 up to
66.2 and 57.8 in correcting annotations on TACRED and DocRED respectively.
These improvements are not merely theoretical: Rather, automatically denoised
training sets demonstrate up to 3.6% performance improvement for
state-of-the-art relation extraction models.
Related papers
- Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation.
We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation.
We then implement several methods for automatically detecting localized factual inconsistencies.
arXiv Detail & Related papers (2024-10-09T22:53:48Z) - Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice.
We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
arXiv Detail & Related papers (2023-12-27T09:03:43Z) - Unsupervised Pretraining for Fact Verification by Language Model
Distillation [4.504050940874427]
We propose SFAVEL (Self-supervised Fact Verification via Language Model Distillation), a novel unsupervised pretraining framework.
It distils self-supervised features into high-quality claim-fact alignments without the need for annotations.
This is enabled by a novel contrastive loss function that encourages features to attain high-quality claim and evidence alignments.
arXiv Detail & Related papers (2023-09-28T15:53:44Z) - Interpretable Automatic Fine-grained Inconsistency Detection in Text
Summarization [56.94741578760294]
We propose the task of fine-grained inconsistency detection, the goal of which is to predict the fine-grained types of factual errors in a summary.
Motivated by how humans inspect factual inconsistency in summaries, we propose an interpretable fine-grained inconsistency detection model, FineGrainFact.
arXiv Detail & Related papers (2023-05-23T22:11:47Z) - Mention Annotations Alone Enable Efficient Domain Adaptation for
Coreference Resolution [8.08448832546021]
We show that annotating mentions alone is nearly twice as fast as annotating full coreference chains.
Our approach facilitates annotation-efficient transfer and results in a 7-14% improvement in average F1 without increasing annotator time.
arXiv Detail & Related papers (2022-10-14T07:57:27Z) - End-to-End Label Uncertainty Modeling in Speech Emotion Recognition
using Bayesian Neural Networks and Label Distribution Learning [0.0]
We propose an end-to-end Bayesian neural network capable of being trained on a distribution of annotations to capture the subjectivity-based label uncertainty.
We show that the proposed t-distribution based approach achieves state-of-the-art uncertainty modeling results in speech emotion recognition.
arXiv Detail & Related papers (2022-09-30T12:55:43Z) - Annotation Error Detection: Analyzing the Past and Present for a More
Coherent Future [63.99570204416711]
We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets.
We define a uniform evaluation setup including a new formalization of the annotation error detection task.
We release our datasets and implementations in an easy-to-use and open source software package.
arXiv Detail & Related papers (2022-06-05T22:31:45Z) - Learning to Revise References for Faithful Summarization [10.795263196202159]
We propose a new approach to improve reference quality while retaining all data.
We construct synthetic unsupported alternatives to supported sentences and use contrastive learning to discourage/encourage (un)faithful revisions.
We extract a small corpus from a noisy source--the Electronic Health Record (EHR)--for the task of summarizing a hospital admission from multiple notes.
arXiv Detail & Related papers (2022-04-13T18:54:19Z) - Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning [57.4036085386653]
We show that prompt-based models for sentence pair classification tasks still suffer from a common pitfall of adopting inferences based on lexical overlap.
We then show that adding a regularization that preserves pretraining weights is effective in mitigating this destructive tendency of few-shot finetuning.
arXiv Detail & Related papers (2021-09-09T10:10:29Z) - Annotation Curricula to Implicitly Train Non-Expert Annotators [56.67768938052715]
voluntary studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain.
This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations.
We propose annotation curricula, a novel approach to implicitly train annotators.
arXiv Detail & Related papers (2021-06-04T09:48:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.