BERTese: Learning to Speak to BERT

• URL: http://arxiv.org/abs/2103.05327v2
• Date: Thu, 11 Mar 2021 08:33:01 GMT
• Title: BERTese: Learning to Speak to BERT
• Authors: Adi Haviv, Jonathan Berant and Amir Globerson
• Abstract summary: We propose a method for automatically rewriting queries into "BERTese", a paraphrase query that is directly optimized towards better knowledge extraction. We empirically show our approach outperforms competing baselines, obviating the need for complex pipelines.
• Score: 50.76152500085082
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large pre-trained language models have been shown to encode large amounts of world and commonsense knowledge in their parameters, leading to substantial interest in methods for extracting that knowledge. In past work, knowledge was extracted by taking manually-authored queries and gathering paraphrases for them using a separate pipeline. In this work, we propose a method for automatically rewriting queries into "BERTese", a paraphrase query that is directly optimized towards better knowledge extraction. To encourage meaningful rewrites, we add auxiliary loss functions that encourage the query to correspond to actual language tokens. We empirically show our approach outperforms competing baselines, obviating the need for complex pipelines. Moreover, BERTese provides some insight into the type of language that helps language models perform knowledge extraction.

Related papers

• Knowledge Graph-Enhanced Large Language Models via Path Selection [58.228392005755026]
  Large Language Models (LLMs) have shown unprecedented performance in various real-world applications. LLMs are known to generate factually inaccurate outputs, a.k.a. the hallucination problem. We propose a principled framework KELP with three stages to handle the above problems.
  arXiv  Detail & Related papers  (2024-06-19T21:45:20Z)
• Adapting PromptORE for Modern History: Information Extraction from Hispanic Monarchy Documents of the XVIth Century [2.490441444378203]
  We introduce an adaptation of PromptORE to extract relations from specialized documents, namely digital transcripts of trials from the Spanish Inquisition. Our approach involves fine-tuning transformer models with their pretraining objective on the data they will perform inference. Our results show a substantial improvement in accuracy -up to a 50% improvement with our Biased PromptORE models.
  arXiv  Detail & Related papers  (2024-05-24T13:39:47Z)
• Query Rewriting for Retrieval-Augmented Large Language Models [139.242907155883]
  Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2023-05-23T17:27:50Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)
• Structured prompt interrogation and recursive extraction of semantics (SPIRES): A method for populating knowledge bases using zero-shot learning [1.3963666696384924]
  We present Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES) SPIRES relies on the ability of Large Language Models (LLMs) to perform zero-shot learning (ZSL) and general-purpose vocabularies answering from flexible prompts and return information to a specified schema. Current SPIRES accuracy is comparable to the mid-range of existing Relation Extraction (RE) methods, but has the advantage of easy customization, flexibility, and, crucially, the ability to perform new tasks in the absence of any training data.
  arXiv  Detail & Related papers  (2023-04-05T19:07:04Z)
• Enriching Relation Extraction with OpenIE [70.52564277675056]
  Relation extraction (RE) is a sub-discipline of information extraction (IE) In this work, we explore how recent approaches for open information extraction (OpenIE) may help to improve the task of RE. Our experiments over two annotated corpora, KnowledgeNet and FewRel, demonstrate the improved accuracy of our enriched models.
  arXiv  Detail & Related papers  (2022-12-19T11:26:23Z)
• DictBERT: Dictionary Description Knowledge Enhanced Language Model Pre-training via Contrastive Learning [18.838291575019504]
  Pre-trained language models (PLMs) are shown to be lacking in knowledge when dealing with knowledge driven tasks. We propose textbfDictBERT, a novel approach that enhances PLMs with dictionary knowledge. We evaluate our approach on a variety of knowledge driven and language understanding tasks, including NER, relation extraction, CommonsenseQA, OpenBookQA and GLUE.
  arXiv  Detail & Related papers  (2022-08-01T06:43:19Z)
• Eliciting Knowledge from Language Models for Event Extraction [3.4448178503887807]
  In this paper, we explore to elicit the knowledge from pre-trained language models for event trigger detection and argument extraction. We present various joint trigger/argument prompt methods, which can elicit more complementary knowledge by modeling the interactions between different triggers or arguments. Our approach is superior to the recent advanced methods in the few-shot scenario where only a few samples are used for training.
  arXiv  Detail & Related papers  (2021-09-11T05:16:33Z)
• Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters [52.725200145600624]
  We propose KnowExpert to bypass the retrieval process by injecting prior knowledge into the pre-trained language models with lightweight adapters. Experimental results show that KnowExpert performs comparably with the retrieval-based baselines.
  arXiv  Detail & Related papers  (2021-05-13T12:33:23Z)
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts [33.43689407735244]
  Natural-language prompts have recently been used to coax pretrained language models into performing other AI tasks. We explore the idea of learning prompts by gradient descent. For each task, we optimize a mixture of prompts, learning which prompts are most effective and how to ensemble them.
  arXiv  Detail & Related papers  (2021-04-14T02:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.