Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation

• URL: http://arxiv.org/abs/2404.04445v1
• Date: Fri, 5 Apr 2024 23:12:46 GMT
• Title: Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation
• Authors: Fahmida Alam, Md Asiful Islam, Robert Vacareanu, Mihai Surdeanu,
• Abstract summary: We introduce a meta dataset for few-shot relation extraction. We conduct a comprehensive evaluation of six recent few-shot relation extraction methods. The overall performance on this task is low, indicating substantial need for future research.
• Score: 17.398872494876365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets NYT29 (Takanobu et al., 2019; Nayak and Ng, 2020) and WIKIDATA (Sorokin and Gurevych, 2017) as well as a few-shot form of the TACRED dataset (Sabo et al., 2021). Importantly, all these few-shot datasets were generated under realistic assumptions such as: the test relations are different from any relations a model might have seen before, limited training data, and a preponderance of candidate relation mentions that do not correspond to any of the relations of interest. Using this large resource, we conduct a comprehensive evaluation of six recent few-shot relation extraction methods, and observe that no method comes out as a clear winner. Further, the overall performance on this task is low, indicating substantial need for future research. We release all versions of the data, i.e., both supervised and few-shot, for future research.

Related papers

• Anno-incomplete Multi-dataset Detection [67.69438032767613]
  We propose a novel problem as "-incomplete Multi-dataset Detection" We develop an end-to-end multi-task learning architecture which can accurately detect all the object categories with multiple partially annotated datasets.
  arXiv  Detail & Related papers  (2024-08-29T03:58:21Z)
• Numerical Literals in Link Prediction: A Critical Examination of Models and Datasets [2.5999037208435705]
  Link Prediction models that incorporate numerical literals have shown minor improvements on existing benchmark datasets. It is unclear whether a model is actually better in using numerical literals, or better capable of utilizing the graph structure. We propose a methodology to evaluate LP models that incorporate numerical literals.
  arXiv  Detail & Related papers  (2024-07-25T17:55:33Z)
• Exploring Factual Entailment with NLI: A News Media Study [0.9208007322096533]
  We explore the relationship between factuality and Natural Language Inference (NLI) by introducing FactRel. Our analysis shows that 84% of factually supporting pairs and 63% of undermining factually pairs do not amount to NLI entailment or contradiction. We experiment with models for pairwise classification on the new dataset, and find that in some cases, generating synthetic data with GPT-4 on the basis of the annotated dataset can improve performance.
  arXiv  Detail & Related papers  (2024-06-24T17:47:55Z)
• Relational Extraction on Wikipedia Tables using Convolutional and Memory Networks [6.200672130699805]
  Relation extraction (RE) is the task of extracting relations between entities in text. We introduce a new model consisting of Convolutional Neural Network (CNN) and Bidirectional-Long Short Term Memory (BiLSTM) network to encode entities.
  arXiv  Detail & Related papers  (2023-07-11T22:36:47Z)
• Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking [66.83273589348758]
  Link prediction attempts to predict whether an unseen edge exists based on only a portion of edges of a graph. A flurry of methods have been introduced in recent years that attempt to make use of graph neural networks (GNNs) for this task. New and diverse datasets have also been created to better evaluate the effectiveness of these new models.
  arXiv  Detail & Related papers  (2023-06-18T01:58:59Z)
• PCRED: Zero-shot Relation Triplet Extraction with Potential Candidate Relation Selection and Entity Boundary Detection [11.274924966891842]
  Zero-shot relation triplet extraction (ZeroRTE) aims to extract relation triplets from unstructured texts. Previous state-of-the-art method handles this challenging task by leveraging pretrained language models to generate data as additional training samples. We tackle this task from a new perspective and propose a novel method named PCRED for ZeroRTE with Potential Candidate Relation selection and Entity boundary Detection.
  arXiv  Detail & Related papers  (2022-11-26T04:27:31Z)
• MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extraction [78.61546292830081]
  We construct a large-scale human-annotated ERE dataset MAVEN-ERE with improved annotation schemes. It contains 103,193 event coreference chains, 1,216,217 temporal relations, 57,992 causal relations, and 15,841 subevent relations. Experiments show that ERE on MAVEN-ERE is quite challenging, and considering relation interactions with joint learning can improve performances.
  arXiv  Detail & Related papers  (2022-11-14T13:34:49Z)
• Comparing Test Sets with Item Response Theory [53.755064720563]
  We evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples. We find that Quoref, HellaSwag, and MC-TACO are best suited for distinguishing among state-of-the-art models. We also observe span selection task format, which is used for QA datasets like QAMR or SQuAD2.0, is effective in differentiating between strong and weak models.
  arXiv  Detail & Related papers  (2021-06-01T22:33:53Z)
• Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction [84.64435075778988]
  We propose a general approach to learn relation prototypes from unlabeled texts. We learn relation prototypes as an implicit factor between entities. We conduct experiments on two publicly available datasets: New York Times and Google Distant Supervision.
  arXiv  Detail & Related papers  (2020-11-27T06:21:12Z)
• Relation of the Relations: A New Paradigm of the Relation Extraction Problem [52.21210549224131]
  We propose a new paradigm of Relation Extraction (RE) that considers as a whole the predictions of all relations in the same context. We develop a data-driven approach that does not require hand-crafted rules but learns by itself the relation of relations (RoR) using Graph Neural Networks and a relation matrix transformer. Experiments show that our model outperforms the state-of-the-art approaches by +1.12% on the ACE05 dataset and +2.55% on SemEval 2018 Task 7.2.
  arXiv  Detail & Related papers  (2020-06-05T22:25:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.