A Preliminary Empirical Study on Prompt-based Unsupervised Keyphrase Extraction

• URL: http://arxiv.org/abs/2405.16571v1
• Date: Sun, 26 May 2024 13:37:57 GMT
• Title: A Preliminary Empirical Study on Prompt-based Unsupervised Keyphrase Extraction
• Authors: Mingyang Song, Yi Feng, Liping Jing,
• Abstract summary: We study the effectiveness of different prompts on the keyphrase extraction task to verify the impact of cherry-picked prompts on the performance of extracting keyphrases. Design complex prompts achieve better performance than designing simple prompts when facing long documents.
• Score: 30.624421412309786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained large language models can perform natural language processing downstream tasks by conditioning on human-designed prompts. However, a prompt-based approach often requires "prompt engineering" to design different prompts, primarily hand-crafted through laborious trial and error, requiring human intervention and expertise. It is a challenging problem when constructing a prompt-based keyphrase extraction method. Therefore, we investigate and study the effectiveness of different prompts on the keyphrase extraction task to verify the impact of the cherry-picked prompts on the performance of extracting keyphrases. Extensive experimental results on six benchmark keyphrase extraction datasets and different pre-trained large language models demonstrate that (1) designing complex prompts may not necessarily be more effective than designing simple prompts; (2) individual keyword changes in the designed prompts can affect the overall performance; (3) designing complex prompts achieve better performance than designing simple prompts when facing long documents.

Related papers

• Grammar-Guided Evolutionary Search for Discrete Prompt Optimisation [63.97051732013936]
  We propose an evolutionary search approach to automated discrete prompt optimisation consisting of two phases.<n>In the first phase, grammar-guided genetic programming is invoked to synthesise prompt-creating programmes.<n>In the second phase, local search is applied to explore the neighbourhoods of best-performing programmes.
  arXiv  Detail & Related papers  (2025-07-14T14:34:15Z)
• Generative Compositor for Few-Shot Visual Information Extraction [60.663887314625164]
  We propose a novel generative model, named Generative generative spatialtor, to address the challenge of few-shot VIE. Generative generative spatialtor is a hybrid pointer-generator network that emulates the operations of a compositor by retrieving words from the source text. The proposed method achieves highly competitive results in the full-sample training, while notably outperforms the baseline in the 1-shot, 5-shot, and 10-shot settings.
  arXiv  Detail & Related papers  (2025-03-21T04:56:24Z)
• Why Prompt Design Matters and Works: A Complexity Analysis of Prompt Search Space in LLMs [15.941209553757274]
  We provide a theoretical framework that explains why some prompts succeed while others fail.<n>We analyze the complexity of finding optimal prompts and characterize the size of the prompt space for a given task.<n>Our theory reveals principles behind effective prompt design and shows that naive CoT-using self-guided prompts like "think step by step"-can severely hinder performance.
  arXiv  Detail & Related papers  (2025-03-13T06:11:10Z)
• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Efficient Prompting Methods for Large Language Models: A Survey [50.171011917404485]
  Prompting has become a mainstream paradigm for adapting large language models (LLMs) to specific natural language processing tasks. This approach brings the additional computational burden of model inference and human effort to guide and control the behavior of LLMs. We present the basic concepts of prompting, review the advances for efficient prompting, and highlight future research directions.
  arXiv  Detail & Related papers  (2024-04-01T12:19:08Z)
• Exploring Prompt Engineering Practices in the Enterprise [3.7882262667445734]
  A prompt is a natural language instruction designed to elicit certain behaviour or output from a model. For complex tasks and tasks with specific requirements, prompt design is not trivial. We analyze sessions of prompt editing behavior, categorizing the parts of prompts users iterated on and the types of changes they made.
  arXiv  Detail & Related papers  (2024-03-13T20:32:32Z)
• Enhancing User Interaction in ChatGPT: Characterizing and Consolidating Multiple Prompts for Issue Resolution [5.176434782905268]
  We analyze 686 prompts submitted to resolve issues related to Java and Python programming languages. We can completely consolidate prompts with four gaps (e.g., missing context) and partially consolidate prompts with three gaps (e.g., additional functionality) Our study findings and evidence can - (a) save users time, (b) reduce costs, and (c) increase user satisfaction.
  arXiv  Detail & Related papers  (2024-02-07T04:07:33Z)
• Towards Robust Multimodal Prompting With Missing Modalities [22.176372579439356]
  multimodal prompting introduces learnable missing-aware prompts for all missing modality cases. It lacks robustness in scenarios with different missing modality settings between training and inference. We propose a simple yet effective prompt design to address these challenges.
  arXiv  Detail & Related papers  (2023-12-26T05:43:55Z)
• Demystifying Prompts in Language Models via Perplexity Estimation [109.59105230163041]
  Performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. We show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task.
  arXiv  Detail & Related papers  (2022-12-08T02:21:47Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)
• MetaPrompting: Learning to Learn Better Prompts [52.914694884515534]
  We propose a new soft prompting method called MetaPrompting, which adopts the well-recognized model-agnostic meta-learning algorithm. Extensive experiments show MetaPrompting brings significant improvement on four different datasets.
  arXiv  Detail & Related papers  (2022-09-23T09:01:05Z)
• Instance-aware Prompt Learning for Language Understanding and Generation [49.22899822734549]
  We propose an instance-aware prompt learning method that learns a different prompt for each instance. Our method achieves the state-of-the-art on the SuperGLUE few-shot learning benchmark.
  arXiv  Detail & Related papers  (2022-01-18T17:03:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.