Related papers: An Exploration of Post-Editing Effectiveness in Text Summarization

An Exploration of Post-Editing Effectiveness in Text Summarization

URL: http://arxiv.org/abs/2206.06383v1
Date: Mon, 13 Jun 2022 18:00:02 GMT
Title: An Exploration of Post-Editing Effectiveness in Text Summarization
Authors: Vivian Lai, Alison Smith-Renner, Ke Zhang, Ruijia Cheng, Wenjuan Zhang, Joel Tetreault, Alejandro Jaimes
Abstract summary: "Post-editing" AI-generated text reduces human workload and improves the quality of AI output. We compare post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience. This study sheds valuable insights on when post-editing is useful for text summarization.
Score: 58.99765574294715
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Automatic summarization methods are efficient but can suffer from low quality. In comparison, manual summarization is expensive but produces higher quality. Can humans and AI collaborate to improve summarization performance? In similar text generation tasks (e.g., machine translation), human-AI collaboration in the form of "post-editing" AI-generated text reduces human workload and improves the quality of AI output. Therefore, we explored whether post-editing offers advantages in text summarization. Specifically, we conducted an experiment with 72 participants, comparing post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience on formal (XSum news) and informal (Reddit posts) text. This study sheds valuable insights on when post-editing is useful for text summarization: it helped in some cases (e.g., when participants lacked domain knowledge) but not in others (e.g., when provided summaries include inaccurate information). Participants' different editing strategies and needs for assistance offer implications for future human-AI summarization systems.

Related papers

How Does the Disclosure of AI Assistance Affect the Perceptions of Writing? [29.068596156140913]
We study whether and how the disclosure of the level and type of AI assistance in the writing process would affect people's perceptions of the writing. Our results suggest that disclosing the AI assistance in the writing process, especially if AI has provided assistance in generating new content, decreases the average quality ratings.
arXiv Detail & Related papers (2024-10-06T16:45:33Z)
A Hybrid Strategy for Chat Transcript Summarization [3.0567294793102784]
Sumomarization is the process of condensing a piece of text to fewer sentences, while still preserving its content. This paper presents an indigenously developed hybrid method that combines extractive and abstractive summarization techniques. Tests, evaluations, comparisons, and validation have demonstrated the efficacy of this approach for large-scale deployment of chat transcript summarization.
arXiv Detail & Related papers (2024-02-02T15:44:28Z)
GUMSum: Multi-Genre Data and Evaluation for English Abstractive Summarization [10.609715843964263]
Automatic summarization with pre-trained language models has led to impressively fluent results, but is prone to 'hallucinations' We present GUMSum, a dataset of English summaries in 12 written and spoken genres for evaluation of abstractive summarization.
arXiv Detail & Related papers (2023-06-20T03:21:10Z)
Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback [57.816210168909286]
We leverage recent progress on textual entailment models to address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.
arXiv Detail & Related papers (2023-05-31T21:04:04Z)
Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks [45.550554287918885]
This paper focuses on evaluating the usefulness of text summaries with extrinsic methods. We design three different downstream tasks for extrinsic human evaluation of summaries, i.e., question answering, text classification and text similarity assessment. We find summaries are particularly useful in tasks that rely on an overall judgment of the text, while being less effective for question answering tasks.
arXiv Detail & Related papers (2023-05-24T11:34:39Z)
Active Learning for Abstractive Text Summarization [50.79416783266641]
We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
arXiv Detail & Related papers (2023-01-09T10:33:14Z)
Human-in-the-loop Abstractive Dialogue Summarization [61.4108097664697]
We propose to incorporate different levels of human feedback into the training process. This will enable us to guide the models to capture the behaviors humans care about for summaries.
arXiv Detail & Related papers (2022-12-19T19:11:27Z)
The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
arXiv Detail & Related papers (2022-12-08T23:23:39Z)
Fine-tuning GPT-3 for Russian Text Summarization [77.34726150561087]
This paper showcases ruGPT3 ability to summarize texts, fine-tuning it on the corpora of Russian news with their corresponding human-generated summaries. We evaluate the resulting texts with a set of metrics, showing that our solution can surpass the state-of-the-art model's performance without additional changes in architecture or loss function.
arXiv Detail & Related papers (2021-08-07T19:01:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.