IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models

• URL: http://arxiv.org/abs/2210.14128v1
• Date: Tue, 25 Oct 2022 16:25:00 GMT
• Title: IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models
• Authors: Chenguang Wang, Xiao Liu, Dawn Song
• Abstract summary: We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM) We create an OIE benchmark aiming to fully examine the open relational information present in the pre-trained LMs. Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets.
• Score: 75.48081086368606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM). Recent studies have demonstrated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge. In particular, LMs are able to answer ``fill-in-the-blank'' questions when given a pre-defined relation category. Instead of focusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pre-trained LMs. We accomplish this by turning pre-trained LMs into zero-shot OIE systems. Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets (CaRB and Re-OIE2016) and two new large-scale factual OIE datasets (TAC KBP-OIE and Wikidata-OIE) that we establish via distant supervision. For instance, the zero-shot pre-trained LMs outperform the F1 score of the state-of-the-art supervised OIE methods on our factual OIE datasets without needing to use any training sets. Our code and datasets are available at https://github.com/cgraywang/IELM

Related papers

• LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models [2.1311017627417]
  Knowledge probing evaluates the extent to which a language model (LM) has acquired relational knowledge during its pre-training phase. We present LM-PUB- QUIZ, a Python framework and leaderboard built around the BEAR probing mechanism.
  arXiv  Detail & Related papers  (2024-08-28T11:44:52Z)
• BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models [2.2863439039616127]
  Probing assesses to which degree a language model (LM) has successfully learned relational knowledge during pre-training. Previous approaches rely on the objective function used in pre-training LMs. We propose an approach that uses an LM's inherent ability to estimate the log-likelihood of any given textual statement.
  arXiv  Detail & Related papers  (2024-04-05T14:13:55Z)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models [52.98743860365194]
  We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN) At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents.
  arXiv  Detail & Related papers  (2024-01-02T18:53:13Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• Meet in the Middle: A New Pre-training Paradigm [41.52858444519968]
  Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion. We propose a new pre-training paradigm with techniques that jointly improve the training data efficiency. We show the effectiveness of our pre-training paradigm with extensive experiments on both programming and natural language models.
  arXiv  Detail & Related papers  (2023-03-13T17:17:11Z)
• An Interpretability Evaluation Benchmark for Pre-trained Language Models [37.16893581395874]
  We propose a novel evaluation benchmark providing with both English and Chinese annotated data. It tests LMs abilities in multiple dimensions, i.e., grammar, semantics, knowledge, reasoning and computation. It contains perturbed instances for each original instance, so as to use the rationale consistency under perturbations as the metric for faithfulness.
  arXiv  Detail & Related papers  (2022-07-28T08:28:09Z)
• Entity Cloze By Date: What LMs Know About Unseen Entities [79.34707800653597]
  Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity.
  arXiv  Detail & Related papers  (2022-05-05T17:59:31Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining [51.19885385587916]
  We conduct studies on semi-supervised learning in the task of text classification under the context of large-scale LM pretraining. Our work marks an initial step in understanding the behavior of semi-supervised learning models under the context of large-scale pretraining.
  arXiv  Detail & Related papers  (2020-11-17T13:39:05Z)
• Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms [36.04356511882304]
  Self-supervised learning (SSL) has demonstrated promising results on a wide range of applications. There has not been a clear understanding on what properties of data and tasks render one approach outperforms the other.
  arXiv  Detail & Related papers  (2020-06-19T05:21:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.