R2D2: Robust Data-to-Text with Replacement Detection
- URL: http://arxiv.org/abs/2205.12467v1
- Date: Wed, 25 May 2022 03:29:25 GMT
- Title: R2D2: Robust Data-to-Text with Replacement Detection
- Authors: Linyong Nan, Lorenzo Jaime Yu Flores, Yilun Zhao, Yixin Liu, Luke
Benson, Weijin Zou, Dragomir Radev
- Abstract summary: We introduce R2D2, a training framework that addresses unfaithful Data-to-Text generation.
We argue that the poor entity retrieval capability of D2T systems is one of the primary sources of unfaithfulness.
Our experimental results show that R2D2 systems could effectively mitigate the unfaithful text generation.
- Score: 16.53137103104244
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Unfaithful text generation is a common problem for text generation systems.
In the case of Data-to-Text (D2T) systems, the factuality of the generated text
is particularly crucial for any real-world applications. We introduce R2D2, a
training framework that addresses unfaithful Data-to-Text generation by
training a system both as a generator and a faithfulness discriminator with
additional replacement detection and unlikelihood learning tasks. To facilitate
such training, we propose two methods for sampling unfaithful sentences. We
argue that the poor entity retrieval capability of D2T systems is one of the
primary sources of unfaithfulness, so in addition to the existing metrics, we
further propose NER-based metrics to evaluate the fidelity of D2T generations.
Our experimental results show that R2D2 systems could effectively mitigate the
unfaithful text generation, and they achieve new state-of-the-art results on
FeTaQA, LogicNLG, and ToTTo, all with significant improvements.
Related papers
- Text2Data: Low-Resource Data Generation with Textual Control [104.38011760992637]
Natural language serves as a common and straightforward control signal for humans to interact seamlessly with machines.
We propose Text2Data, a novel approach that utilizes unlabeled data to understand the underlying data distribution through an unsupervised diffusion model.
It undergoes controllable finetuning via a novel constraint optimization-based learning objective that ensures controllability and effectively counteracts catastrophic forgetting.
arXiv Detail & Related papers (2024-02-08T03:41:39Z) - Contrastive Transformer Learning with Proximity Data Generation for
Text-Based Person Search [60.626459715780605]
Given a descriptive text query, text-based person search aims to retrieve the best-matched target person from an image gallery.
Such a cross-modal retrieval task is quite challenging due to significant modality gap, fine-grained differences and insufficiency of annotated data.
In this paper, we propose a simple yet effective dual Transformer model for text-based person search.
arXiv Detail & Related papers (2023-11-15T16:26:49Z) - Successor Features for Efficient Multisubject Controlled Text Generation [48.37713738712319]
We introduce SF-GEN, which is grounded in two primary concepts: successor features (SFs) and language model rectification.
SF-GEN seamlessly integrates the two to enable dynamic steering of text generation with no need to alter the LLM's parameters.
To the best of our knowledge, our research represents the first application of successor features in text generation.
arXiv Detail & Related papers (2023-11-03T00:17:08Z) - RADAR: Robust AI-Text Detection via Adversarial Learning [69.5883095262619]
RADAR is based on adversarial training of a paraphraser and a detector.
The paraphraser's goal is to generate realistic content to evade AI-text detection.
RADAR uses the feedback from the detector to update the paraphraser, and vice versa.
arXiv Detail & Related papers (2023-07-07T21:13:27Z) - On the Possibilities of AI-Generated Text Detection [76.55825911221434]
We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases.
We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
arXiv Detail & Related papers (2023-04-10T17:47:39Z) - SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic
Mistakes [93.19166902594168]
We propose SESCORE2, a self-supervised approach for training a model-based metric for text generation evaluation.
Key concept is to synthesize realistic model mistakes by perturbing sentences retrieved from a corpus.
We evaluate SESCORE2 and previous methods on four text generation tasks across three languages.
arXiv Detail & Related papers (2022-12-19T09:02:16Z) - A Benchmark Corpus for the Detection of Automatically Generated Text in
Academic Publications [0.02578242050187029]
This paper presents two datasets comprised of artificially generated research content.
In the first case, the content is completely generated by the GPT-2 model after a short prompt extracted from original papers.
The partial or hybrid dataset is created by replacing several sentences of abstracts with sentences that are generated by the Arxiv-NLP model.
We evaluate the quality of the datasets comparing the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE.
arXiv Detail & Related papers (2022-02-04T08:16:56Z) - Evaluating Semantic Accuracy of Data-to-Text Generation with Natural
Language Inference [3.42658286826597]
A major challenge in evaluating data-to-text (D2T) generation is measuring the semantic accuracy of the generated text.
We propose a new metric for evaluating the semantic accuracy of D2T generation based on a neural model pretrained for natural language inference (NLI)
Our experiments on two recent D2T datasets show that our metric can achieve high accuracy in identifying erroneous system outputs.
arXiv Detail & Related papers (2020-11-21T16:37:28Z) - Tweet to News Conversion: An Investigation into Unsupervised
Controllable Text Generation [46.74654716230366]
In this paper, we define the task of constructing a coherent paragraph from a set of disaster domain tweets.
We tackle the problem by building two systems in pipeline. The first system focuses on unsupervised style transfer and converts the individual tweets into news sentences.
The second system stitches together the outputs from the first system to form a coherent news paragraph.
arXiv Detail & Related papers (2020-08-21T06:56:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.