Position-based Prompting for Health Outcome Generation
- URL: http://arxiv.org/abs/2204.03489v1
- Date: Wed, 30 Mar 2022 16:44:04 GMT
- Title: Position-based Prompting for Health Outcome Generation
- Authors: M. Abaho, D. Bollegala, P. Williamson, S. Dodd
- Abstract summary: We explore an idea of using a position-attention mechanism to capture positional information of each word in a prompt relative to the mask to be filled.
Our approach consistently outperforms a baseline in which the default mask language model (MLM) representation is used to predict masked tokens.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Probing Pre-trained Language Models (PLMs) using prompts has indirectly
implied that language models (LMs) can be treated as knowledge bases. To this
end, this phenomena has been effective especially when these LMs are fine-tuned
towards not just data of a specific domain, but also to the style or linguistic
pattern of the prompts themselves. We observe that, satisfying a particular
linguistic pattern in prompts is an unsustainable constraint that unnecessarily
lengthens the probing task, especially because, they are often manually
designed and the range of possible prompt template patterns can vary depending
on the prompting objective and domain. We therefore explore an idea of using a
position-attention mechanism to capture positional information of each word in
a prompt relative to the mask to be filled, hence avoiding the need to
re-construct prompts when the prompts linguistic pattern changes. Using our
approach, we demonstrate the ability of eliciting answers to rare prompt
templates (in a case study on health outcome generation) such as Postfix and
Mixed patterns whose missing information is respectively at the start and in
multiple random places of the prompt. More so, using various biomedical PLMs,
our approach consistently outperforms a baseline in which the default mask
language model (MLM) representation is used to predict masked tokens.
Related papers
- Eliciting Textual Descriptions from Representations of Continuous Prompts [11.489611613744724]
We propose a new approach to interpret continuous prompts that elicits textual descriptions from their representations during model inference.
We show our method often yields accurate task descriptions which become more faithful as task performance increases.
InSPEcT can be leveraged to debug unwanted properties in continuous prompts and inform developers on ways to mitigate them.
arXiv Detail & Related papers (2024-10-15T14:46:11Z) - Leveraging Hallucinations to Reduce Manual Prompt Dependency in Promptable Segmentation [74.04806143723597]
We introduce an iterative Prompt-Mask Cycle generation framework (ProMaC) with a prompt generator and a mask generator.
The prompt generator uses a multi-scale chain of thought prompting, initially exploring hallucinations for extracting extended contextual knowledge on a test image.
The generated masks iteratively induce the prompt generator to focus more on task-relevant image areas and reduce irrelevant hallucinations, resulting jointly in better prompts and masks.
arXiv Detail & Related papers (2024-08-27T17:06:22Z) - Helping Language Models Learn More: Multi-dimensional Task Prompt for
Few-shot Tuning [36.14688633670085]
We propose MTPrompt, a multi-dimensional task prompt learning method based on task-related object, summary, and task description information.
By automatically building and searching for appropriate prompts, our proposed MTPrompt achieves the best results on few-shot samples setting and five different datasets.
arXiv Detail & Related papers (2023-12-13T10:00:44Z) - Eliciting Human Preferences with Language Models [56.68637202313052]
Language models (LMs) can be directed to perform target tasks by using labeled examples or natural language prompts.
We propose to use *LMs themselves* to guide the task specification process.
We study GATE in three domains: email validation, content recommendation, and moral reasoning.
arXiv Detail & Related papers (2023-10-17T21:11:21Z) - Bayesian Prompt Learning for Image-Language Model Generalization [64.50204877434878]
We use the regularization ability of Bayesian methods to frame prompt learning as a variational inference problem.
Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts.
We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space.
arXiv Detail & Related papers (2022-10-05T17:05:56Z) - Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified
Multilingual Prompt [98.26682501616024]
We propose a novel model that uses a unified prompt for all languages, called UniPrompt.
The unified prompt is computation by a multilingual PLM to produce language-independent representation.
Our proposed methods can significantly outperform the strong baselines across different languages.
arXiv Detail & Related papers (2022-02-23T11:57:52Z) - Context-Tuning: Learning Contextualized Prompts for Natural Language
Generation [52.835877179365525]
We propose a novel continuous prompting approach, called Context-Tuning, to fine-tuning PLMs for natural language generation.
Firstly, the prompts are derived based on the input text, so that they can elicit useful knowledge from PLMs for generation.
Secondly, to further enhance the relevance of the generated text to the inputs, we utilize continuous inverse prompting to refine the process of natural language generation.
arXiv Detail & Related papers (2022-01-21T12:35:28Z) - AutoPrompt: Eliciting Knowledge from Language Models with Automatically
Generated Prompts [46.03503882865222]
AutoPrompt is an automated method to create prompts for a diverse set of tasks based on a gradient-guided search.
We show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning.
arXiv Detail & Related papers (2020-10-29T22:54:00Z) - UniLMv2: Pseudo-Masked Language Models for Unified Language Model
Pre-Training [152.63467944568094]
We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks.
Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks.
arXiv Detail & Related papers (2020-02-28T15:28:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.