An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models

• URL: http://arxiv.org/abs/2109.02772v1
• Date: Mon, 6 Sep 2021 23:29:36 GMT
• Title: An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models
• Authors: Tianxing He, Kyunghyun Cho, James Glass
• Abstract summary: We show that few-shot examples can strongly boost the probing performance for both 1-hop and 2-hop relations. In particular, we find that a simple-yet-effective approach of finetuning the bias vectors in the model outperforms existing prompt-engineering methods.
• Score: 54.74525882974022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt-based knowledge probing for 1-hop relations has been used to measure how much world knowledge is stored in pretrained language models. Existing work uses considerable amounts of data to tune the prompts for better performance. In this work, we compare a variety of approaches under a few-shot knowledge probing setting, where only a small number (e.g., 10 or 20) of example triples are available. In addition, we create a new dataset named TREx-2p, which contains 2-hop relations. We report that few-shot examples can strongly boost the probing performance for both 1-hop and 2-hop relations. In particular, we find that a simple-yet-effective approach of finetuning the bias vectors in the model outperforms existing prompt-engineering methods. Our dataset and code are available at \url{https://github.com/cloudygoose/fewshot_lama}.

Related papers

• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• How to Unleash the Power of Large Language Models for Few-shot Relation Extraction? [28.413620806193165]
  In this paper, we investigate principal methodologies, in-context learning and data generation, for few-shot relation extraction via GPT-3.5. We observe that in-context learning can achieve performance on par with previous prompt learning approaches, and data generation with the large language model can boost previous solutions to obtain new state-of-the-art few-shot results.
  arXiv  Detail & Related papers  (2023-05-02T15:55:41Z)
• UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph [89.98762327725112]
  Multi-hop Question Answering over Knowledge Graph(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question. We propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning.
  arXiv  Detail & Related papers  (2022-12-02T04:08:09Z)
• Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning [109.7767515627765]
  We propose a new semiparametric paradigm of retrieval-enhanced prompt tuning for relation extraction. Our model infers relation through knowledge stored in the weights during training. Our method can achieve state-of-the-art in both standard supervised and few-shot settings.
  arXiv  Detail & Related papers  (2022-05-04T23:38:37Z)
• PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models [67.3725459417758]
  PERFECT is a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting. We show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning. Experiments on a wide range of few-shot NLP tasks demonstrate that PERFECT, while being simple and efficient, also outperforms existing state-of-the-art few-shot learning methods.
  arXiv  Detail & Related papers  (2022-04-03T22:31:25Z)
• Factual Probing Is [MASK]: Learning vs. Learning to Recall [8.668111159444273]
  Petroni et al. demonstrated that it is possible to retrieve world facts from a pre-trained language model by expressing them as cloze-style prompts. We make two complementary contributions to better understand these factual probing techniques. We find, somewhat surprisingly, that the training data used by these methods contains certain regularities of the underlying fact distribution.
  arXiv  Detail & Related papers  (2021-04-12T07:11:40Z)
• ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning [10.609715843964263]
  We formulate the zero-shot relation extraction problem by incorporating the text description of seen and unseen relations. We propose a novel multi-task learning model, zero-shot BERT, to directly predict unseen relations without hand-crafted labeling and multiple pairwise attribute classifications. Experiments conducted on two well-known datasets exhibit that ZS-BERT can outperform existing methods by at least 13.54% improvement on F1 score.
  arXiv  Detail & Related papers  (2021-04-10T06:53:41Z)
• Deep Indexed Active Learning for Matching Heterogeneous Entity Representations [20.15233789156307]
  We propose DIAL, a scalable active learning approach that jointly learns embeddings to maximize recall for blocking and accuracy for matching blocked pairs. Experiments on five benchmark datasets and a multilingual record matching dataset show the effectiveness of our approach in terms of precision, recall and running time.
  arXiv  Detail & Related papers  (2021-04-08T18:00:19Z)
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation [100.79870384880333]
  We propose a knowledge-grounded pre-training (KGPT) to generate knowledge-enriched text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under zero-shot setting, our model achieves over 30 ROUGE-L on WebNLG while all other baselines fail.
  arXiv  Detail & Related papers  (2020-10-05T19:59:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.