Can discrete information extraction prompts generalize across language models?

• URL: http://arxiv.org/abs/2302.09865v1
• Date: Mon, 20 Feb 2023 09:56:51 GMT
• Title: Can discrete information extraction prompts generalize across language models?
• Authors: Nathana\"el Carraz Rakotonirina, Roberto Dess\`i, Fabio Petroni, Sebastian Riedel, Marco Baroni
• Abstract summary: We study whether automatically-induced prompts can also be used, out-of-the-box, to probe other language models for the same information. We introduce a way to induce prompts by mixing language models at training time that results in prompts that generalize well across models.
• Score: 36.85568212975316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study whether automatically-induced prompts that effectively extract information from a language model can also be used, out-of-the-box, to probe other language models for the same information. After confirming that discrete prompts induced with the AutoPrompt algorithm outperform manual and semi-manual prompts on the slot-filling task, we demonstrate a drop in performance for AutoPrompt prompts learned on a model and tested on another. We introduce a way to induce prompts by mixing language models at training time that results in prompts that generalize well across models. We conduct an extensive analysis of the induced prompts, finding that the more general prompts include a larger proportion of existing English words and have a less order-dependent and more uniform distribution of information across their component tokens. Our work provides preliminary evidence that it's possible to generate discrete prompts that can be induced once and used with a number of different models, and gives insights on the properties characterizing such prompts.

Related papers

• OLaLa: Ontology Matching with Large Language Models [2.211868306499727]
  Ontology Matching is a challenging task where information in natural language is one of the most important signals to process. With the rise of Large Language Models, it is possible to incorporate this knowledge in a better way into the matching pipeline. We show that with only a handful of examples and a well-designed prompt, it is possible to achieve results that are en par with supervised matching systems.
  arXiv  Detail & Related papers  (2023-11-07T09:34:20Z)
• Effective Prompt Extraction from Language Models [78.67410369494623]
  We present a framework for measuring the effectiveness of prompt extraction attacks. In experiments with 3 different sources of prompts and 11 underlying large language models, we find that simple text-based attacks can in fact reveal prompts with high probability. Our framework determines with high precision whether an extracted prompt is the actual secret prompt, rather than a model hallucination.
  arXiv  Detail & Related papers  (2023-07-13T16:15:08Z)
• Boosted Prompt Ensembles for Large Language Models [38.402161594793775]
  Methods such as chain-of-thought prompting and self-consistency have pushed the frontier of language model reasoning performance with no additional training. We propose a prompt ensembling method for large language models, which uses a small dataset to construct a set of few shot prompts that together comprise a boosted prompt ensemble'' We show that this outperforms single-prompt output-space ensembles and bagged prompt-space ensembles on the GSM8k and AQuA datasets.
  arXiv  Detail & Related papers  (2023-04-12T16:47:15Z)
• Demystifying Prompts in Language Models via Perplexity Estimation [100.43627541756524]
  Performance of a prompt is coupled with the extent to which the model is familiar with the language it contains. We show that the lower the perplexity of the prompt is, the better the prompt is able to perform the task.
  arXiv  Detail & Related papers  (2022-12-08T02:21:47Z)
• Explaining Patterns in Data with Language Models via Interpretable Autoprompting [143.4162028260874]
  We introduce interpretable autoprompting (iPrompt), an algorithm that generates a natural-language string explaining the data. iPrompt can yield meaningful insights by accurately finding groundtruth dataset descriptions. Experiments with an fMRI dataset show the potential for iPrompt to aid in scientific discovery.
  arXiv  Detail & Related papers  (2022-10-04T18:32:14Z)
• Probing via Prompting [71.7904179689271]
  This paper introduces a novel model-free approach to probing, by formulating probing as a prompting task. We conduct experiments on five probing tasks and show that our approach is comparable or better at extracting information than diagnostic probes. We then examine the usefulness of a specific linguistic property for pre-training by removing the heads that are essential to that property and evaluating the resulting model's performance on language modeling.
  arXiv  Detail & Related papers  (2022-07-04T22:14:40Z)
• Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt [98.26682501616024]
  We propose a novel model that uses a unified prompt for all languages, called UniPrompt. The unified prompt is computation by a multilingual PLM to produce language-independent representation. Our proposed methods can significantly outperform the strong baselines across different languages.
  arXiv  Detail & Related papers  (2022-02-23T11:57:52Z)
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm [0.0]
  We discuss methods of prompt programming, emphasizing the usefulness of considering prompts through the lens of natural language. We introduce the idea of a metaprompt that seeds the model to generate its own natural language prompts for a range of tasks.
  arXiv  Detail & Related papers  (2021-02-15T05:27:55Z)
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts [46.03503882865222]
  AutoPrompt is an automated method to create prompts for a diverse set of tasks based on a gradient-guided search. We show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning.
  arXiv  Detail & Related papers  (2020-10-29T22:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.