Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance

• URL: http://arxiv.org/abs/2412.17321v1
• Date: Mon, 23 Dec 2024 06:29:25 GMT
• Title: Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance
• Authors: Nicolas Devatine, Louis Abraham,
• Abstract summary: Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing. We introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm.
• Score: 2.1792283995628465
• License:
• Abstract: Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing, especially when edits involve substantial modifications, such as block operations. In this paper, we introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm, designed to quantify the amount of post-editing applied to LLM-generated texts. Our method leverages the properties of text compression to measure the informational difference between the original and edited texts. Through experiments on real-world human edits datasets, we demonstrate that our proposed metric is highly correlated with actual edit time and effort. We also show that LLMs exhibit an implicit understanding of editing speed, that aligns well with our metric. Furthermore, we compare our metric with existing ones, highlighting its advantages in capturing complex edits with linear computational efficiency. Our code and data are available at: https://github.com/NDV-tiime/CompressionDistance

Related papers

• Bridging the Editing Gap in LLMs: FineEdit for Precise and Targeted Text Modifications [9.795246551841586]
  Large Language Models (LLMs) have transformed natural language processing, yet they still struggle with direct text editing tasks. In this work, we introduce a dual approach to enhance LLM editing performance. First, we present InstrEditBench, a high-quality benchmark dataset comprising over 20,000 structured editing tasks. Second, we propose FineEdit, a specialized model trained on this curated benchmark.
  arXiv  Detail & Related papers  (2025-02-19T01:41:44Z)
• Edit Distances and Their Applications to Downstream Tasks in Research and Commercial Contexts [7.629053304626553]
  The tutorial describes the concept of edit distances applied to research and commercial contexts. We use Translation Edit Rate (TER), Levenshtein, Damerau-Levenshtein, Longest Common Subsequence and $n$-gram distances to demonstrate the frailty of statistical metrics when comparing text sequences.
  arXiv  Detail & Related papers  (2024-10-08T10:21:22Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update. We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• ParSEL: Parameterized Shape Editing with Language [17.312928067096543]
  ParSEL is a system that enables controllable editing of high-quality 3D assets from natural language. adjusting the program parameters allows users to explore shape variations with a precise control over the magnitudes of edits.
  arXiv  Detail & Related papers  (2024-05-30T17:55:46Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach [50.400999859808984]
  Post-editing has proven effective in improving the quality of text generated by large language models (LLMs) We propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs when editing their output. Experiments demonstrate that the programmer-interpreter significantly enhances GPT-3.5's performance in logical form-to-text conversion and low-resource machine translation.
  arXiv  Detail & Related papers  (2024-02-07T06:13:14Z)
• WikiIns: A High-Quality Dataset for Controlled Text Editing by Natural Language Instruction [56.196512595940334]
  We build and release WikiIns, a high-quality controlled text editing dataset with improved informativeness. With the high-quality annotated dataset, we propose automatic approaches to generate a large-scale silver'' training set.
  arXiv  Detail & Related papers  (2023-10-08T04:46:39Z)
• XATU: A Fine-grained Instruction-based Benchmark for Explainable Text Updates [7.660511135287692]
  This paper introduces XATU, the first benchmark specifically designed for fine-grained instruction-based explainable text editing. XATU considers finer-grained text editing tasks of varying difficulty, incorporating lexical, syntactic, semantic, and knowledge-intensive edit aspects. We demonstrate the effectiveness of instruction tuning and the impact of underlying architecture across various editing tasks.
  arXiv  Detail & Related papers  (2023-09-20T04:58:59Z)
• Reducing Sequence Length by Predicting Edit Operations with Large Language Models [50.66922361766939]
  This paper proposes predicting edit spans for the source text for local sequence transduction tasks. We apply instruction tuning for Large Language Models on the supervision data of edit spans. Experiments show that the proposed method achieves comparable performance to the baseline in four tasks.
  arXiv  Detail & Related papers  (2023-05-19T17:51:05Z)
• Text Editing by Command [82.50904226312451]
  A prevailing paradigm in neural text generation is one-shot generation, where text is produced in a single step. We address this limitation with an interactive text generation setting in which the user interacts with the system by issuing commands to edit existing text. We show that our Interactive Editor, a transformer-based model trained on this dataset, outperforms baselines and obtains positive results in both automatic and human evaluations.
  arXiv  Detail & Related papers  (2020-10-24T08:00:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.