BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion

• URL: http://arxiv.org/abs/2603.11415v1
• Date: Thu, 12 Mar 2026 01:14:19 GMT
• Title: BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion
• Abstract summary: We propose BLooP, a simple training-free decoding intervention that encourages large language models to generate bigrams from the source document.<n>BLooP operates through a hash table lookup at each decoding step, requiring no training, fine-tuning, or model modification.<n>Human evaluation shows that BLooP significantly improves faithfulness without reducing readability.
• Score: 52.26069987232466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Abstractive summarization requires models to generate summaries that convey information in the source document. While large language models can generate summaries without fine-tuning, they often miss key details and include extraneous information. We propose BLooP (Bigram Lookahead Promotion), a simple training-free decoding intervention that encourages large language models (LLMs) to generate tokens that form bigrams from the source document. BLooP operates through a hash table lookup at each decoding step, requiring no training, fine-tuning, or model modification. We demonstrate improvements in ROUGE and BARTScore for Llama-3.1-8B-Instruct, Mistral-Nemo-Instruct-2407, and Gemma-2-9b-it on CNN/DM, CCSum, Multi-News, and SciTLDR. Human evaluation shows that BLooP significantly improves faithfulness without reducing readability. We make the code available at https://github.com/varuniyer/BLooP

Related papers

• Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarization [10.892363692521222]
  Abstractive summarization aims to generate concise summaries by creating new sentences, allowing for flexible rephrasing.<n>This approach can be vulnerable to inaccuracies, particularly hallucinations' where the model introduces non-existent information.<n>We leverage the use of multimodal and multilingual sentence embeddings derived from pretrained models such as LaBSE, SONAR, and BGE-M3, and feed them into a modified BART-based French model.<n>A Named Entity Injection mechanism that appends tokenized named entities to the decoder input is introduced, in order to improve the factual consistency of the generated summary.
  arXiv  Detail & Related papers  (2026-03-09T11:55:32Z)
• TLDR: Token-Level Detective Reward Model for Large Vision Language Models [57.41524422460438]
  Existing reward models only mimic human annotations by assigning only one binary feedback to any text.<n>We propose a $textbfT$oken-$textbfL$evel $textbfD$etective $textbfR$eward Model to provide fine-grained annotations to each text token.
  arXiv  Detail & Related papers  (2024-10-07T04:00:22Z)
• GLIMMER: Incorporating Graph and Lexical Features in Unsupervised Multi-Document Summarization [13.61818620609812]
  We propose a lightweight yet effective unsupervised approach called GLIMMER: a Graph and LexIcal features based unsupervised Multi-docuMEnt summaRization approach. It first constructs a sentence graph from the source documents, then automatically identifies semantic clusters by mining low-level features from raw texts. Experiments conducted on Multi-News, Multi-XScience and DUC-2004 demonstrate that our approach outperforms existing unsupervised approaches.
  arXiv  Detail & Related papers  (2024-08-19T16:01:48Z)
• TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Data [50.40191599304911]
  This paper proposes a simple but effective framework: Transliterate-Merge-literation (TransMI)<n>TransMI can create strong baselines for data that is transliterated into a common script by exploiting an existing mPLM and its tokenizer without any training.<n>Our experiments demonstrate that TransMI not only preserves the mPLM's ability to handle non-transliterated data, but also enables it to effectively process transliterated data, thereby facilitating crosslingual transfer across scripts.
  arXiv  Detail & Related papers  (2024-05-16T09:08:09Z)
• Read between the lines -- Functionality Extraction From READMEs [7.302971710984904]
  We introduce a novel and useful variant of it called functionality extraction from Git files. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.
  arXiv  Detail & Related papers  (2024-03-15T11:11:57Z)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models [121.0693322732454]
  This paper proposes a textbfCraFT' approach for fine-tuning black-box vision-language models to downstream tasks. CraFT comprises two modules, a prompt generation module for learning text prompts and a prediction refinement module for enhancing output predictions in residual style. Experiments on few-shot classification over 15 datasets demonstrate the superiority of CraFT.
  arXiv  Detail & Related papers  (2024-02-06T14:53:19Z)
• Generating EDU Extracts for Plan-Guided Summary Re-Ranking [77.7752504102925]
  Two-step approaches, in which summary candidates are generated-then-reranked to return a single summary, can improve ROUGE scores over the standard single-step approach. We design a novel method to generate candidates for re-ranking that addresses these issues. We show large relevance improvements over previously published methods on widely used single document news article corpora.
  arXiv  Detail & Related papers  (2023-05-28T17:22:04Z)
• Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentation [42.05617728412819]
  We show how to optimize few-shot text classification without accessing the gradients of the large-scale language models. Our approach, dubbed BT-Classifier, significantly outperforms state-of-the-art black-box few-shot learners.
  arXiv  Detail & Related papers  (2023-05-23T07:54:34Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees [89.60269205320431]
  Current abstractive summarization models either suffer from a lack of clear interpretability or provide incomplete rationales. We propose the Summarization Program (SP), an interpretable modular framework consisting of an (ordered) list of binary trees. A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence to the document sentences.
  arXiv  Detail & Related papers  (2022-09-21T16:50:22Z)
• Meta Learning for Code Summarization [10.403206672504664]
  We show that three SOTA models for code summarization work well on largely disjoint subsets of a large code-base. We propose three meta-models that select the best candidate summary for a given code segment.
  arXiv  Detail & Related papers  (2022-01-20T17:23:34Z)
• Fact-level Extractive Summarization with Hierarchical Graph Mask on BERT [9.271716501646194]
  We propose to extract fact-level semantic units for better extractive summarization. We incorporate our model with BERT using a hierarchical graph mask. Experiments on the CNN/DaliyMail dataset show that our model achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2020-11-19T09:29:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.