Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models

• URL: http://arxiv.org/abs/2310.06374v2
• Date: Sun, 22 Oct 2023 08:37:43 GMT
• Title: Rethinking Model Selection and Decoding for Keyphrase Generation with Pre-trained Sequence-to-Sequence Models
• Authors: Di Wu, Wasi Uddin Ahmad, Kai-Wei Chang
• Abstract summary: Keyphrase Generation (KPG) is a longstanding task in NLP with widespread applications. Seq2seq pre-trained language models (PLMs) have ushered in a transformative era for KPG, yielding promising performance improvements. This paper undertakes a systematic analysis of the influence of model selection and decoding strategies on PLM-based KPG.
• Score: 76.52997424694767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Keyphrase Generation (KPG) is a longstanding task in NLP with widespread applications. The advent of sequence-to-sequence (seq2seq) pre-trained language models (PLMs) has ushered in a transformative era for KPG, yielding promising performance improvements. However, many design decisions remain unexplored and are often made arbitrarily. This paper undertakes a systematic analysis of the influence of model selection and decoding strategies on PLM-based KPG. We begin by elucidating why seq2seq PLMs are apt for KPG, anchored by an attention-driven hypothesis. We then establish that conventional wisdom for selecting seq2seq PLMs lacks depth: (1) merely increasing model size or performing task-specific adaptation is not parameter-efficient; (2) although combining in-domain pre-training with task adaptation benefits KPG, it does partially hinder generalization. Regarding decoding, we demonstrate that while greedy search achieves strong F1 scores, it lags in recall compared with sampling-based methods. Based on these insights, we propose DeSel, a likelihood-based decode-select algorithm for seq2seq PLMs. DeSel improves greedy search by an average of 4.7% semantic F1 across five datasets. Our collective findings pave the way for deeper future investigations into PLM-based KPG.

Related papers

• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function [50.812404038684505]
  Reinforcement Learning From Human Feedback (RLHF) has been a critical to the success of the latest generation of generative AI models. Direct Preference Optimization (DPO) has emerged as an alternative approach. DPO solves the same objective as the standard RLHF setup, but there is a mismatch between the two approaches.
  arXiv  Detail & Related papers  (2024-04-18T17:37:02Z)
• On Leveraging Encoder-only Pre-trained Language Models for Effective Keyphrase Generation [76.52997424694767]
  This study addresses the application of encoder-only Pre-trained Language Models (PLMs) in keyphrase generation (KPG) With encoder-only PLMs, although KPE with Conditional Random Fields slightly excels in identifying present keyphrases, the KPG formulation renders a broader spectrum of keyphrase predictions. We also identify a favorable parameter allocation towards model depth rather than width when employing encoder-decoder architectures with encoder-only PLMs.
  arXiv  Detail & Related papers  (2024-02-21T18:57:54Z)
• ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph [142.42275983201978]
  We propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning. We also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data.
  arXiv  Detail & Related papers  (2023-12-30T07:18:54Z)
• Ranking-based Adaptive Query Generation for DETRs in Crowded Pedestrian Detection [49.27380156754935]
  We find that the number of DETRs' queries must be adjusted manually, otherwise, the performance would degrade to varying degrees. We propose Rank-based Adaptive Query Generation (RAQG) to alleviate the problem. Our method is simple and effective, which can be plugged into any DETRs to make it query-adaptive in theory.
  arXiv  Detail & Related papers  (2023-10-24T11:00:56Z)
• Stopping Criteria for Value Iteration on Stochastic Games with Quantitative Objectives [0.0]
  A classic solution technique for Markov decision processes (MDP) and games (SG) is value (VI) In this paper, we provide the first stopping criteria for VI on SG with total reward and mean payoff, yielding the first anytime algorithms in these settings.
  arXiv  Detail & Related papers  (2023-04-19T19:09:55Z)
• An Empirical Study of Pre-trained Language Models in Simple Knowledge Graph Question Answering [28.31377197194905]
  Large-scale pre-trained language models (PLMs) have recently achieved great success and become a milestone in natural language processing (NLP) In recent works on knowledge graph question answering (KGQA), BERT or its variants have become necessary in their KGQA models. We compare the performance of different PLMs in KGQA and present three benchmarks for larger-scale KGs.
  arXiv  Detail & Related papers  (2023-03-18T08:57:09Z)
• GROOT: Corrective Reward Optimization for Generative Sequential Labeling [10.306943706927004]
  We propose GROOT -- a framework for Generative Reward Optimization Of Text sequences. GROOT works by training a generative sequential labeling model to match the decoder output distribution with that of the (black-box) reward function. As demonstrated via extensive experiments on four public benchmarks, GROOT significantly improves all reward metrics.
  arXiv  Detail & Related papers  (2022-09-29T11:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.