Edisum: Summarizing and Explaining Wikipedia Edits at Scale

• URL: http://arxiv.org/abs/2404.03428v2
• Date: Sun, 18 Aug 2024 16:22:58 GMT
• Title: Edisum: Summarizing and Explaining Wikipedia Edits at Scale
• Authors: Marija Šakota, Isaac Johnson, Guosheng Feng, Robert West,
• Abstract summary: We propose a model for recommending edit summaries generated by a language model trained to produce good edit summaries. Our model performs on par with human editors. More broadly, we showcase how language modeling technology can be used to support humans in maintaining one of the largest and most visible projects on the Web.
• Score: 9.968020416365757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An edit summary is a succinct comment written by a Wikipedia editor explaining the nature of, and reasons for, an edit to a Wikipedia page. Edit summaries are crucial for maintaining the encyclopedia: they are the first thing seen by content moderators and they help them decide whether to accept or reject an edit. Additionally, edit summaries constitute a valuable data source for researchers. Unfortunately, as we show, for many edits, summaries are either missing or incomplete. To overcome this problem and help editors write useful edit summaries, we propose a model for recommending edit summaries generated by a language model trained to produce good edit summaries given the representation of an edit diff. To overcome the challenges of mixed-quality training data and efficiency requirements imposed by the scale of Wikipedia, we fine-tune a small generative language model on a curated mix of human and synthetic data. Our model performs on par with human editors. Commercial large language models are able to solve this task better than human editors, but are not well suited for Wikipedia, while open-source ones fail on this task. More broadly, we showcase how language modeling technology can be used to support humans in maintaining one of the largest and most visible projects on the Web.

Related papers

• Should We Really Edit Language Models? On the Evaluation of Edited Language Models [15.63231238452797]
  Existing editing methods lead to inevitable performance deterioration on general benchmarks. Instruction-tuned models are more robust to editing, showing less performance drop on general knowledge after editing. Our findings indicate that current editing methods are only suitable for small-scale knowledge updates within language models.
  arXiv  Detail & Related papers  (2024-10-24T14:36:48Z)
• Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs? [61.68363765350178]
  This paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing research. We first describe 12 open problems with model editing, based on challenges with (1) defining the problem, (2) developing benchmarks, and (3) assuming LLMs have editable beliefs in the first place. Next, we introduce a semi-synthetic dataset for model editing based on Wikidata, where we can evaluate edits against labels given by an idealized Bayesian agent.
  arXiv  Detail & Related papers  (2024-06-27T17:33:03Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• DUnE: Dataset for Unified Editing [3.7346004746366384]
  We introduce DUnE-an editing benchmark where edits are natural language sentences. We show that retrieval-augmented language modeling can outperform specialized editing techniques.
  arXiv  Detail & Related papers  (2023-11-27T18:56:14Z)
• WikiIns: A High-Quality Dataset for Controlled Text Editing by Natural Language Instruction [56.196512595940334]
  We build and release WikiIns, a high-quality controlled text editing dataset with improved informativeness. With the high-quality annotated dataset, we propose automatic approaches to generate a large-scale silver'' training set.
  arXiv  Detail & Related papers  (2023-10-08T04:46:39Z)
• SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages [87.08880616654258]
  We introduce the SWiPE dataset, which reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) articles. We work with Wikipedia editors to annotate 5,000 EW-SEW document pairs, labeling more than 40,000 edits with proposed 19 categories. We find that SWiPE-trained models generate more complex edits while reducing unwanted edits.
  arXiv  Detail & Related papers  (2023-05-30T16:52:42Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)
• Learning Structural Edits via Incremental Tree Transformations [102.64394890816178]
  We present a generic model for incremental editing of structured data (i.e., "structural edits") Our editor learns to iteratively generate tree edits (e.g., deleting or adding a subtree) and applies them to the partially edited data. We evaluate our proposed editor on two source code edit datasets, where results show that, with the proposed edit encoder, our editor significantly improves accuracy over previous approaches.
  arXiv  Detail & Related papers  (2021-01-28T16:11:32Z)
• Scalable Recommendation of Wikipedia Articles to Editors Using Representation Learning [1.8810916321241067]
  We develop a scalable system on top of Graph Convolutional Networks and Doc2Vec, learning how to represent Wikipedia articles and deliver personalized recommendations for editors. We test our model on editors' histories, predicting their most recent edits based on their prior edits. All of the data used on this paper is publicly available, including graph embeddings for Wikipedia articles, and we release our code to support replication of our experiments.
  arXiv  Detail & Related papers  (2020-09-24T15:56:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.