Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

• URL: http://arxiv.org/abs/2401.10415v2
• Date: Thu, 27 Jun 2024 04:00:19 GMT
• Title: Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?
• Authors: Marcio Fonseca, Shay B. Cohen,
• Abstract summary: We investigate the controllability of large language models (LLMs) on scientific summarization tasks. We find that non-fine-tuned LLMs outperform humans in the MuP review generation task.
• Score: 19.814974042343028
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-fine-tuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly fine-tuning remains an open problem for domain-specific applications.

Related papers

• Zero-to-Strong Generalization: Eliciting Strong Capabilities of Large Language Models Iteratively without Gold Labels [75.77877889764073]
  Large Language Models (LLMs) have demonstrated remarkable performance through supervised fine-tuning or in-context learning using gold labels. This study explores whether solely utilizing unlabeled data can elicit strong model capabilities. We propose a new paradigm termed zero-to-strong generalization.
  arXiv  Detail & Related papers  (2024-09-19T02:59:44Z)
• Leveraging the Power of LLMs: A Fine-Tuning Approach for High-Quality Aspect-Based Summarization [25.052557735932535]
  Large language models (LLMs) have demonstrated the potential to revolutionize diverse tasks within natural language processing. This paper explores the potential of fine-tuning LLMs for the aspect-based summarization task. We evaluate the impact of fine-tuning open-source foundation LLMs, including Llama2, Mistral, Gemma and Aya, on a publicly available domain-specific aspect based summary dataset.
  arXiv  Detail & Related papers  (2024-08-05T16:00:21Z)
• Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models [59.970391602080205]
  This study investigates whether such constraints on generation space impact LLMs abilities, including reasoning and domain knowledge comprehension. We evaluate LLMs performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. We find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.
  arXiv  Detail & Related papers  (2024-08-05T13:08:24Z)
• Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistency [5.9858789096400224]
  Large language models (LLMs) suffer from factual inconsistency problem called hallucinations. We present a novel summary generation strategy, namely SliSum, which exploits the ideas of sliding windows and self-consistency. SliSum significantly improves the faithfulness of diverse LLMs including LLaMA-2, Claude-2 and GPT-3.5 in both short and long text summarization.
  arXiv  Detail & Related papers  (2024-07-31T08:48:48Z)
• Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasing [37.400757839157116]
  Large Language Models (LLMs) have achieved state-of-the-art performance at zero-shot generation of abstractive summaries for given articles. We propose relevance paraphrasing, a simple strategy that can be used to measure the robustness of LLMs as summarizers.
  arXiv  Detail & Related papers  (2024-06-06T12:08:43Z)
• Unveiling the Generalization Power of Fine-Tuned Large Language Models [81.70754292058258]
  We investigate whether fine-tuning affects the intrinsic generalization ability intrinsic to Large Language Models (LLMs) Our main findings reveal that models fine-tuned on generation and classification tasks exhibit dissimilar behaviors in generalizing to different domains and tasks. We observe that integrating the in-context learning strategy during fine-tuning on generation tasks can enhance the model's generalization ability.
  arXiv  Detail & Related papers  (2024-03-14T08:18:59Z)
• Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy [48.29181662640212]
  Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. We consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs.
  arXiv  Detail & Related papers  (2024-02-20T08:41:23Z)
• Summarization is (Almost) Dead [49.360752383801305]
  We develop new datasets and conduct human evaluation experiments to evaluate the zero-shot generation capability of large language models (LLMs) Our findings indicate a clear preference among human evaluators for LLM-generated summaries over human-written summaries and summaries generated by fine-tuned models.
  arXiv  Detail & Related papers  (2023-09-18T08:13:01Z)
• On Learning to Summarize with Large Language Models as References [101.79795027550959]
  Large language models (LLMs) are favored by human annotators over the original reference summaries in commonly used summarization datasets. We study an LLM-as-reference learning setting for smaller text summarization models to investigate whether their performance can be substantially improved.
  arXiv  Detail & Related papers  (2023-05-23T16:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.