Weakly- and Semi-supervised Evidence Extraction
- URL: http://arxiv.org/abs/2011.01459v1
- Date: Tue, 3 Nov 2020 04:05:00 GMT
- Title: Weakly- and Semi-supervised Evidence Extraction
- Authors: Danish Pruthi, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton
- Abstract summary: We propose new methods to combine few evidence annotations with abundant document-level labels for the task of evidence extraction.
Our approach yields substantial gains with as few as hundred evidence annotations.
- Score: 107.47661281843232
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: For many prediction tasks, stakeholders desire not only predictions but also
supporting evidence that a human can use to verify its correctness. However, in
practice, additional annotations marking supporting evidence may only be
available for a minority of training examples (if available at all). In this
paper, we propose new methods to combine few evidence annotations (strong
semi-supervision) with abundant document-level labels (weak supervision) for
the task of evidence extraction. Evaluating on two classification tasks that
feature evidence annotations, we find that our methods outperform baselines
adapted from the interpretability literature to our task. Our approach yields
substantial gains with as few as hundred evidence annotations. Code and
datasets to reproduce our work are available at
https://github.com/danishpruthi/evidence-extraction.
Related papers
- Search, Examine and Early-Termination: Fake News Detection with Annotation-Free Evidences [32.11238363508177]
We propose an approach named textbfSEE that retrieves useful information from web-searched annotation-free evidences.
The experimental results demonstrate that the proposed method outperforms state-of-the-art approaches.
arXiv Detail & Related papers (2024-07-10T07:22:30Z) - Salience Allocation as Guidance for Abstractive Summarization [61.31826412150143]
We propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON)
SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness.
arXiv Detail & Related papers (2022-10-22T02:13:44Z) - Annotation Error Detection: Analyzing the Past and Present for a More
Coherent Future [63.99570204416711]
We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets.
We define a uniform evaluation setup including a new formalization of the annotation error detection task.
We release our datasets and implementations in an easy-to-use and open source software package.
arXiv Detail & Related papers (2022-06-05T22:31:45Z) - Fine-Grained Visual Entailment [51.66881737644983]
We propose an extension of this task, where the goal is to predict the logical relationship of fine-grained knowledge elements within a piece of text to an image.
Unlike prior work, our method is inherently explainable and makes logical predictions at different levels of granularity.
We evaluate our method on a new dataset of manually annotated knowledge elements and show that our method achieves 68.18% accuracy at this challenging task.
arXiv Detail & Related papers (2022-03-29T16:09:38Z) - Distantly-Supervised Evidence Retrieval Enables Question Answering
without Evidence Annotation [19.551623461082617]
Open-domain question answering answers a question based on evidence retrieved from a large corpus.
This paper investigates whether models can learn to find evidence from a large corpus, with only distant supervision from answer labels for model training.
We introduce a novel approach (DistDR) that iteratively improves over a weak retriever by alternately finding evidence from the up-to-date model and encouraging the model to learn the most likely evidence.
arXiv Detail & Related papers (2021-10-10T20:01:27Z) - Assisting the Human Fact-Checkers: Detecting All Previously Fact-Checked
Claims in a Document [27.076320857009655]
Given an input document, it aims to detect all sentences that contain a claim that can be verified by some previously fact-checked claims.
The output is a re-ranked list of the document sentences, so that those that can be verified are ranked as high as possible.
Our analysis demonstrates the importance of modeling text similarity and stance, while also taking into account the veracity of the retrieved previously fact-checked claims.
arXiv Detail & Related papers (2021-09-14T13:46:52Z) - Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
We propose methods to facilitate identification of training data artifacts.
We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data.
We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
arXiv Detail & Related papers (2021-07-01T09:26:13Z) - SueNes: A Weakly Supervised Approach to Evaluating Single-Document
Summarization via Negative Sampling [25.299937353444854]
We present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries.
Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries.
arXiv Detail & Related papers (2020-05-13T15:40:13Z) - Unsupervised Opinion Summarization with Noising and Denoising [85.49169453434554]
We create a synthetic dataset from a corpus of user reviews by sampling a review, pretending it is a summary, and generating noisy versions thereof.
At test time, the model accepts genuine reviews and generates a summary containing salient opinions, treating those that do not reach consensus as noise.
arXiv Detail & Related papers (2020-04-21T16:54:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.