Incremental Beam Manipulation for Natural Language Generation

• URL: http://arxiv.org/abs/2102.02574v1
• Date: Thu, 4 Feb 2021 12:26:47 GMT
• Title: Incremental Beam Manipulation for Natural Language Generation
• Authors: James Hargreaves, Andreas Vlachos, Guy Emerson
• Abstract summary: It is common to rerank the output of beam search, but this relies on beam search to produce a good set of hypotheses. Other alternatives to beam search require changes to the training of the model, which restricts their applicability. This paper proposes incremental beam manipulation, i.e. reranking the hypotheses in the beam during decoding instead of only at the end.
• Score: 26.295452668557452
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of natural language generation systems has improved substantially with modern neural networks. At test time they typically employ beam search to avoid locally optimal but globally suboptimal predictions. However, due to model errors, a larger beam size can lead to deteriorating performance according to the evaluation metric. For this reason, it is common to rerank the output of beam search, but this relies on beam search to produce a good set of hypotheses, which limits the potential gains. Other alternatives to beam search require changes to the training of the model, which restricts their applicability compared to beam search. This paper proposes incremental beam manipulation, i.e. reranking the hypotheses in the beam during decoding instead of only at the end. This way, hypotheses that are unlikely to lead to a good final output are discarded, and in their place hypotheses that would have been ignored will be considered instead. Applying incremental beam manipulation leads to an improvement of 1.93 and 5.82 BLEU points over vanilla beam search for the test sets of the E2E and WebNLG challenges respectively. The proposed method also outperformed a strong reranker by 1.04 BLEU points on the E2E challenge, while being on par with it on the WebNLG dataset.

Related papers

• Semantic-guided Search for Efficient Program Repair with Large Language Models [0.9319432628663639]
  FLAMES employs semantic-guided patch generation to enhance repair effectiveness and memory efficiency. FLAMES substantially reduces memory consumption by up to 83% compared to conventional LLM-based APR. Remarkably, FLAMES successfully generated 133 and 103 correct fixes for 333 and 163 bugs in the Defects4J and HumanEval-Java datasets.
  arXiv  Detail & Related papers  (2024-10-22T02:59:47Z)
• Self-Evaluation Guided Beam Search for Reasoning [61.523627290397556]
  We introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of Large Language Model (LLM) We propose a decoding algorithm integrating the self-evaluation guidance via beam search. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by $6.34%$, $9.56%$, and $5.46%$ on the GSM8K, AQuA, and StrategyQA.
  arXiv  Detail & Related papers  (2023-05-01T02:37:59Z)
• Improving Dual-Encoder Training through Dynamic Indexes for Negative Mining [61.09807522366773]
  We introduce an algorithm that approximates the softmax with provable bounds and that dynamically maintains the tree. In our study on datasets with over twenty million targets, our approach cuts error by half in relation to oracle brute-force negative mining.
  arXiv  Detail & Related papers  (2023-03-27T15:18:32Z)
• A Call for Clarity in Beam Search: How It Works and When It Stops [125.55175954381991]
  We introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search. Empirical results demonstrate that adjusting this patience factor improves decoding performance of strong pretrained models on news text summarization and machine translation over diverse language pairs.
  arXiv  Detail & Related papers  (2022-04-11T22:03:44Z)
• Sampling-Based Minimum Bayes Risk Decoding for Neural Machine Translation [20.76001576262768]
  We show that a sampling-based approximation to minimum Bayes risk (MBR) decoding has no equivalent to the beam search curse. We also show that it can be beneficial to make use of strategies like beam search and nucleus sampling to construct hypothesis spaces efficiently.
  arXiv  Detail & Related papers  (2021-08-10T14:35:24Z)
• An Empirical Investigation of Beam-Aware Training in Supertagging [29.819517845454815]
  Structured prediction is often approached by training a locally normalized model with maximum likelihood and decoding approximately with beam search. Beam-aware training aims to address these problems, but it is not yet widely used due to a lack of understanding about how it impacts performance.
  arXiv  Detail & Related papers  (2020-10-10T12:25:18Z)
• If beam search is the answer, what was the question? [78.71330480725668]
  We find that beam search enforces uniform information density in text, a property motivated by cognitive science. We suggest a set of decoding objectives that explicitly enforce this property and find that exact decoding with these objectives alleviates the problems encountered when decoding poorly calibrated language generation models.
  arXiv  Detail & Related papers  (2020-10-06T11:57:03Z)
• Best-First Beam Search [78.71330480725668]
  We show that the standard implementation of beam search can be made up to 10x faster in practice. We propose a memory-reduced variant of Best-First Beam Search, which has a similar beneficial search bias in terms of downstream performance.
  arXiv  Detail & Related papers  (2020-07-08T05:56:01Z)
• L2R2: Leveraging Ranking for Abductive Reasoning [65.40375542988416]
  The abductive natural language inference task ($alpha$NLI) is proposed to evaluate the abductive reasoning ability of a learning system. A novel $L2R2$ approach is proposed under the learning-to-rank framework. Experiments on the ART dataset reach the state-of-the-art in the public leaderboard.
  arXiv  Detail & Related papers  (2020-05-22T15:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.