Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts

• URL: http://arxiv.org/abs/2305.15689v2
• Date: Sat, 1 Jul 2023 23:02:18 GMT
• Title: Zero-shot Approach to Overcome Perturbation Sensitivity of Prompts
• Authors: Mohna Chakraborty, Adithya Kulkarni, Qi Li
• Abstract summary: Recent studies have demonstrated that natural-language prompts can help to leverage the knowledge learned by pre-trained language models for the binary sentence-level sentiment classification task. This study aims to find high-quality prompts for the given task in a zero-shot setting. We empirically demonstrate that the top-ranked prompts are high-quality and significantly outperform the base prompt and the prompts generated using few-shot learning for the binary sentence-level sentiment classification task.
• Score: 7.208567411886273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent studies have demonstrated that natural-language prompts can help to leverage the knowledge learned by pre-trained language models for the binary sentence-level sentiment classification task. Specifically, these methods utilize few-shot learning settings to fine-tune the sentiment classification model using manual or automatically generated prompts. However, the performance of these methods is sensitive to the perturbations of the utilized prompts. Furthermore, these methods depend on a few labeled instances for automatic prompt generation and prompt ranking. This study aims to find high-quality prompts for the given task in a zero-shot setting. Given a base prompt, our proposed approach automatically generates multiple prompts similar to the base prompt employing positional, reasoning, and paraphrasing techniques and then ranks the prompts using a novel metric. We empirically demonstrate that the top-ranked prompts are high-quality and significantly outperform the base prompt and the prompts generated using few-shot learning for the binary sentence-level sentiment classification task.

Related papers

• Automated Few-shot Classification with Instruction-Finetuned Language Models [76.69064714392165]
  We show that AuT-Few outperforms state-of-the-art few-shot learning methods. We also show that AuT-Few is the best ranking method across datasets on the RAFT few-shot benchmark.
  arXiv  Detail & Related papers  (2023-05-21T21:50:27Z)
• A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models [30.128204719490856]
  We aim to automate prompt engineering and improve zero-shot accuracy through prompt ensembling. We identify several pathologies in a naive prompt scoring method where the score can be easily overconfident due to biases in pre-training and test data. Using our proposed scoring method to create a weighted average prompt ensemble, our method outperforms equal average ensemble, as well as hand-crafted prompts.
  arXiv  Detail & Related papers  (2023-02-13T10:19:58Z)
• Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too? [84.91689960190054]
  Large language models can perform new tasks in a zero-shot fashion, given natural language prompts. It is underexplored what factors make the prompts effective, especially when the prompts are natural language.
  arXiv  Detail & Related papers  (2022-12-20T18:47:13Z)
• Demystifying Prompts in Language Models via Perplexity Estimation [109.59105230163041]
  Performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. We show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task.
  arXiv  Detail & Related papers  (2022-12-08T02:21:47Z)
• TEMPERA: Test-Time Prompting via Reinforcement Learning [57.48657629588436]
  We propose Test-time Prompt Editing using Reinforcement learning (TEMPERA) In contrast to prior prompt generation methods, TEMPERA can efficiently leverage prior knowledge. Our method achieves 5.33x on average improvement in sample efficiency when compared to the traditional fine-tuning methods.
  arXiv  Detail & Related papers  (2022-11-21T22:38:20Z)
• MetaPrompting: Learning to Learn Better Prompts [52.914694884515534]
  We propose a new soft prompting method called MetaPrompting, which adopts the well-recognized model-agnostic meta-learning algorithm. Extensive experiments show MetaPrompting brings significant improvement on four different datasets.
  arXiv  Detail & Related papers  (2022-09-23T09:01:05Z)
• Instance-aware Prompt Learning for Language Understanding and Generation [49.22899822734549]
  We propose an instance-aware prompt learning method that learns a different prompt for each instance. Our method achieves the state-of-the-art on the SuperGLUE few-shot learning benchmark.
  arXiv  Detail & Related papers  (2022-01-18T17:03:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.