Can GPT models Follow Human Summarization Guidelines? A Study for Targeted Communication Goals

• URL: http://arxiv.org/abs/2310.16810v2
• Date: Mon, 07 Apr 2025 21:42:15 GMT
• Title: Can GPT models Follow Human Summarization Guidelines? A Study for Targeted Communication Goals
• Authors: Yongxin Zhou, Fabien Ringeval, François Portet,
• Abstract summary: This study investigates the ability of GPT models to generate dialogue summaries that adhere to human guidelines.<n>Our findings reveal a preference for GPT-generated summaries over those from task-specific pre-trained models and reference summaries.<n>The discrepancy between ROUGE, BERTScore, and human evaluation underscores the need for more reliable automatic evaluation metrics.
• Score: 3.875021622948646
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This study investigates the ability of GPT models (ChatGPT, GPT-4 and GPT-4o) to generate dialogue summaries that adhere to human guidelines. Our evaluation involved experimenting with various prompts to guide the models in complying with guidelines on two datasets: DialogSum (English social conversations) and DECODA (French call center interactions). Human evaluation, based on summarization guidelines, served as the primary assessment method, complemented by extensive quantitative and qualitative analyses. Our findings reveal a preference for GPT-generated summaries over those from task-specific pre-trained models and reference summaries, highlighting GPT models' ability to follow human guidelines despite occasionally producing longer outputs and exhibiting divergent lexical and structural alignment with references. The discrepancy between ROUGE, BERTScore, and human evaluation underscores the need for more reliable automatic evaluation metrics.

Related papers

• Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization [0.0]
  This paper delves into the reproduction and evaluation of dialogue summarization models. Our research involved a thorough examination of several dialogue summarization models using the AMI dataset. The primary objective was to evaluate the informativeness and quality of the summaries generated by these models through human assessment.
  arXiv  Detail & Related papers  (2024-10-21T12:47:57Z)
• Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT [0.6740832660968358]
  This research examines the effectiveness of OpenAI's GPT models as independent evaluators of text summaries generated by six transformer-based models. We evaluated these summaries based on essential properties of high-quality summary - conciseness, relevance, coherence, and readability - using traditional metrics such as ROUGE and Latent Semantic Analysis (LSA) Our analysis revealed significant correlations between GPT and traditional metrics, particularly in assessing relevance and coherence.
  arXiv  Detail & Related papers  (2024-05-07T06:52:34Z)
• Information-Theoretic Distillation for Reference-less Summarization [67.51150817011617]
  We present a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization. We start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization. We arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT.
  arXiv  Detail & Related papers  (2024-03-20T17:42:08Z)
• AugSumm: towards generalizable speech summarization using synthetic labels from large language model [61.73741195292997]
  Abstractive speech summarization (SSUM) aims to generate human-like summaries from speech. conventional SSUM models are mostly trained and evaluated with a single ground-truth (GT) human-annotated deterministic summary. We propose AugSumm, a method to leverage large language models (LLMs) as a proxy for human annotators to generate augmented summaries.
  arXiv  Detail & Related papers  (2024-01-10T18:39:46Z)
• CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation [87.44350003888646]
  Eval-Instruct can acquire pointwise grading critiques with pseudo references and revise these critiques via multi-path prompting. CritiqueLLM is empirically shown to outperform ChatGPT and all the open-source baselines.
  arXiv  Detail & Related papers  (2023-11-30T16:52:42Z)
• Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks [9.801767683867125]
  We provide a preliminary and hybrid evaluation on three NLP benchmarks using both automatic and human evaluation. We find that ChatGPT consistently outperforms many other popular models according to human reviewers on the majority of metrics. We also find that human reviewers rate the gold reference as much worse than the best models' outputs, indicating the poor quality of many popular benchmarks.
  arXiv  Detail & Related papers  (2023-10-20T20:17:09Z)
• On the Generalization of Training-based ChatGPT Detection Methods [33.46128880100525]
  ChatGPT is one of the most popular language models which achieve amazing performance on various natural language tasks. There is also an urgent need to detect the texts generated ChatGPT from human written.
  arXiv  Detail & Related papers  (2023-10-02T16:13:08Z)
• Learning Evaluation Models from Large Language Models for Sequence Generation [61.8421748792555]
  We propose a three-stage evaluation model training method that utilizes large language models to generate labeled data for model-based metric development. Experimental results on the SummEval benchmark demonstrate that CSEM can effectively train an evaluation model without human-labeled data.
  arXiv  Detail & Related papers  (2023-08-08T16:41:16Z)
• Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect ChatGPT-Generated Text [48.36706154871577]
  We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts) It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts. We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
  arXiv  Detail & Related papers  (2023-07-21T06:38:37Z)
• ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfer [8.64514166615844]
  We conduct a systematic inspection of ChatGPT's performance in two controllable generation tasks. We evaluate the faithfulness of the generated text, and compare the model's performance with human-authored texts. We observe that ChatGPT sometimes incorporates factual errors or hallucinations when adapting the text to suit a specific style.
  arXiv  Detail & Related papers  (2023-06-13T14:21:35Z)
• On the Detectability of ChatGPT Content: Benchmarking, Methodology, and Evaluation through the Lens of Academic Writing [10.534162347659514]
  We develop a deep neural framework named CheckGPT to better capture the subtle and deep semantic and linguistic patterns in ChatGPT written literature. To evaluate the detectability of ChatGPT content, we conduct extensive experiments on the transferability, prompt engineering, and robustness of CheckGPT.
  arXiv  Detail & Related papers  (2023-06-07T12:33:24Z)
• Comparing Abstractive Summaries Generated by ChatGPT to Real Summaries Through Blinded Reviewers and Text Classification Algorithms [0.8339831319589133]
  ChatGPT, developed by OpenAI, is a recent addition to the family of language models. We evaluate the performance of ChatGPT on Abstractive Summarization by the means of automated metrics and blinded human reviewers.
  arXiv  Detail & Related papers  (2023-03-30T18:28:33Z)
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment [64.01972723692587]
  We present G-Eval, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm to assess the quality of NLG outputs. We show that G-Eval with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin.
  arXiv  Detail & Related papers  (2023-03-29T12:46:54Z)
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study [121.77986688862302]
  We provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.
  arXiv  Detail & Related papers  (2023-03-07T16:57:20Z)
• Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization [28.104696513516117]
  Large language models (LLMs) like GPT3 and ChatGPT have recently created significant interest in using these models for text summarization tasks. Recent studies citegoyal2022news, zhang2023benchmarking have shown that LLMs-generated news summaries are already on par with humans. Our experiments reveal that ChatGPT's performance is comparable to traditional fine-tuning methods in terms of Rouge scores.
  arXiv  Detail & Related papers  (2023-02-16T04:41:30Z)
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation [136.16507050034755]
  Existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale. We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units. We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems.
  arXiv  Detail & Related papers  (2022-12-15T17:26:05Z)
• News Summarization and Evaluation in the Era of GPT-3 [73.48220043216087]
  We study how GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, prompted using only a task description, but these also do not suffer from common dataset-specific issues such as poor factuality.
  arXiv  Detail & Related papers  (2022-09-26T01:04:52Z)
• SummEval: Re-evaluating Summarization Evaluation [169.622515287256]
  We re-evaluate 14 automatic evaluation metrics in a comprehensive and consistent fashion. We benchmark 23 recent summarization models using the aforementioned automatic evaluation metrics. We assemble the largest collection of summaries generated by models trained on the CNN/DailyMail news dataset.
  arXiv  Detail & Related papers  (2020-07-24T16:25:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.