FRUIT: Faithfully Reflecting Updated Information in Text

• URL: http://arxiv.org/abs/2112.08634v1
• Date: Thu, 16 Dec 2021 05:21:24 GMT
• Title: FRUIT: Faithfully Reflecting Updated Information in Text
• Authors: Robert L. Logan IV, Alexandre Passos, Sameer Singh and Ming-Wei Chang
• Abstract summary: We introduce the novel generation task of *faithfully reflecting updated information in text*(FRUIT) Our analysis shows that developing models that can update articles faithfully requires new capabilities for neural generation models.
• Score: 106.40177769765512
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Textual knowledge bases such as Wikipedia require considerable effort to keep up to date and consistent. While automated writing assistants could potentially ease this burden, the problem of suggesting edits grounded in external knowledge has been under-explored. In this paper, we introduce the novel generation task of *faithfully reflecting updated information in text*(FRUIT) where the goal is to update an existing article given new evidence. We release the FRUIT-WIKI dataset, a collection of over 170K distantly supervised data produced from pairs of Wikipedia snapshots, along with our data generation pipeline and a gold evaluation set of 914 instances whose edits are guaranteed to be supported by the evidence. We provide benchmark results for popular generation systems as well as EDIT5 -- a T5-based approach tailored to editing we introduce that establishes the state of the art. Our analysis shows that developing models that can update articles faithfully requires new capabilities for neural generation models, and opens doors to many new applications.

Related papers

• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Language Modeling with Editable External Knowledge [90.7714362827356]
  This paper introduces ERASE, which improves model behavior when new documents are acquired. It incrementally deletes or rewriting other entries in the knowledge base each time a document is added. It improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute.
  arXiv  Detail & Related papers  (2024-06-17T17:59:35Z)
• Detecting Edited Knowledge in Language Models [5.260519479124422]
  Knowledge editing methods (KEs) can update language models' obsolete or inaccurate knowledge learned from pre-training. Knowing whether a generated output is based on edited knowledge or first-hand knowledge from pre-training can increase users' trust in generative models. We propose a novel task: detecting edited knowledge in language models.
  arXiv  Detail & Related papers  (2024-05-04T22:02:24Z)
• Grounded Keys-to-Text Generation: Towards Factual Open-Ended Generation [92.1582872870226]
  We propose a new grounded keys-to-text generation task. The task is to generate a factual description about an entity given a set of guiding keys, and grounding passages. Inspired by recent QA-based evaluation measures, we propose an automatic metric, MAFE, for factual correctness of generated descriptions.
  arXiv  Detail & Related papers  (2022-12-04T23:59:41Z)
• IRJIT: A Simple, Online, Information Retrieval Approach for Just-In-Time Software Defect Prediction [10.084626547964389]
  Just-in-Time software defect prediction (JIT-SDP) prevents the introduction of defects into the software by identifying them at commit check-in time. Current software defect prediction approaches rely on manually crafted features such as change metrics and involve expensive to train machine learning or deep learning models. We propose an approach called IRJIT that employs information retrieval on source code and labels new commits as buggy or clean based on their similarity to past buggy or clean commits.
  arXiv  Detail & Related papers  (2022-10-05T17:54:53Z)
• EditEval: An Instruction-Based Benchmark for Text Improvements [73.5918084416016]
  This work presents EditEval: An instruction-based, benchmark and evaluation suite for automatic evaluation of editing capabilities. We evaluate several pre-trained models, which shows that InstructGPT and PEER perform the best, but that most baselines fall below the supervised SOTA. Our analysis shows that commonly used metrics for editing tasks do not always correlate well, and that optimization for prompts with the highest performance does not necessarily entail the strongest robustness to different models.
  arXiv  Detail & Related papers  (2022-09-27T12:26:05Z)
• Focused Attention Improves Document-Grounded Generation [111.42360617630669]
  Document grounded generation is the task of using the information provided in a document to improve text generation. This work focuses on two different document grounded generation tasks: Wikipedia Update Generation task and Dialogue response generation.
  arXiv  Detail & Related papers  (2021-04-26T16:56:29Z)
• Revisiting Challenges in Data-to-Text Generation with Fact Grounding [2.969705152497174]
  We introduce a larger-scale dataset, RotoWire-FG (Ground-Facting), with 50% more data from the year 2017-19. We achieve improved data fidelity over the state-of-the-art models by integrating a new form of table reconstruction.
  arXiv  Detail & Related papers  (2020-01-12T02:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.