Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits

• URL: http://arxiv.org/abs/2409.14509v3
• Date: Thu, 26 Sep 2024 03:15:53 GMT
• Title: Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits
• Authors: Tuhin Chakrabarty, Philippe Laban, Chien-Sheng Wu,
• Abstract summary: We hired professional writers to edit paragraphs in several creative domains. We curated the LAMP corpus: 1,057 LLM-generated paragraphs edited by professional writers according to our taxonomy. Analysis of LAMP reveals that none of the LLMs used in our study outperform each other in terms of writing quality.
• Score: 39.00434175773803
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based applications are helping people write, and LLM-generated text is making its way into social media, journalism, and our classrooms. However, the differences between LLM-generated and human-written text remain unclear. To explore this, we hired professional writers to edit paragraphs in several creative domains. We first found these writers agree on undesirable idiosyncrasies in LLM-generated text, formalizing it into a seven-category taxonomy (e.g. cliches, unnecessary exposition). Second, we curated the LAMP corpus: 1,057 LLM-generated paragraphs edited by professional writers according to our taxonomy. Analysis of LAMP reveals that none of the LLMs used in our study (GPT4o, Claude-3.5-Sonnet, Llama-3.1-70b) outperform each other in terms of writing quality, revealing common limitations across model families. Third, we explored automatic editing methods to improve LLM-generated text. A large-scale preference annotation confirms that although experts largely prefer text edited by other experts, automatic editing methods show promise in improving alignment between LLM-generated and human-written text.

Related papers

• Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection [7.242609314791262]
  Human & LLM Paraphrase Collection (HLPC) is a first-of-its-kind dataset that incorporates human-written texts and paraphrases. We perform classification experiments that incorporate human-written paraphrases, watermarked and non-watermarked LLM-generated documents from GPT and OPT, and LLM-generated paraphrases from DIPPER and BART. Results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.
  arXiv  Detail & Related papers  (2024-11-06T10:06:21Z)
• Customizing Large Language Model Generation Style using Parameter-Efficient Finetuning [24.263699489328427]
  One-size-fits-all large language models (LLMs) are increasingly being used to help people with their writing. This paper explores whether parameter-efficient finetuning (PEFT) with Low-Rank Adaptation can effectively guide the style of LLM generations.
  arXiv  Detail & Related papers  (2024-09-06T19:25:18Z)
• Learning to Rewrite: Generalized LLM-Generated Text Detection [19.9477991969521]
  Large language models (LLMs) can be abused at scale to create non-factual content and spread disinformation. We propose training an LLM to rewrite input text, producing minimal edits for LLM-generated content and more edits for human-written text. Our work suggests that LLM can effectively detect machine-generated text if they are trained properly.
  arXiv  Detail & Related papers  (2024-08-08T05:53:39Z)
• LLM-Collaboration on Automatic Science Journalism for the General Audience [3.591143309194537]
  Science journalism reports current scientific discoveries to non-specialists. This task can be challenging as the audience often lacks specific knowledge about the presented research. We propose a framework that integrates three LLMs mimicking the real-world writing-reading-feedback-revision workflow.
  arXiv  Detail & Related papers  (2024-07-13T03:31:35Z)
• ReMoDetect: Reward Models Recognize Aligned LLM's Generations [55.06804460642062]
  Large language models (LLMs) generate human-preferable texts. In this paper, we identify the common characteristics shared by these models. We propose two training schemes to further improve the detection ability of the reward model.
  arXiv  Detail & Related papers  (2024-05-27T17:38:33Z)
• Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach [50.400999859808984]
  Post-editing has proven effective in improving the quality of text generated by large language models (LLMs) We propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs when editing their output. Experiments demonstrate that the programmer-interpreter significantly enhances GPT-3.5's performance in logical form-to-text conversion and low-resource machine translation.
  arXiv  Detail & Related papers  (2024-02-07T06:13:14Z)
• LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback [65.84061725174269]
  Recent large language models (LLM) are leveraging human feedback to improve their generation quality. We propose LLMRefine, an inference time optimization method to refine LLM's output. We conduct experiments on three text generation tasks, including machine translation, long-form question answering (QA), and topical summarization. LLMRefine consistently outperforms all baseline approaches, achieving improvements up to 1.7 MetricX points on translation tasks, 8.1 ROUGE-L on ASQA, 2.2 ROUGE-L on topical summarization.
  arXiv  Detail & Related papers  (2023-11-15T19:52:11Z)
• Improving Factual Consistency of Text Summarization by Adversarially Decoupling Comprehension and Embellishment Abilities of LLMs [67.56087611675606]
  Large language models (LLMs) generate summaries that are factually inconsistent with original articles. These hallucinations are challenging to detect through traditional methods. We propose an adversarially DEcoupling method to disentangle the abilities of LLMs (DECENT)
  arXiv  Detail & Related papers  (2023-10-30T08:40:16Z)
• Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases [0.0]
  Large Language Models (LLMs) have posed a serious threat to academic integrity in education. Modern detectors are still in need of improvements so that they can offer a full-proof solution to help maintain academic integrity.
  arXiv  Detail & Related papers  (2023-07-10T12:18:34Z)
• Red Teaming Language Model Detectors with Language Models [114.36392560711022]
  Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
  arXiv  Detail & Related papers  (2023-05-31T10:08:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.