CrossRE: A Cross-Domain Dataset for Relation Extraction
- URL: http://arxiv.org/abs/2210.09345v1
- Date: Mon, 17 Oct 2022 18:33:14 GMT
- Title: CrossRE: A Cross-Domain Dataset for Relation Extraction
- Authors: Elisa Bassignana and Barbara Plank
- Abstract summary: CrossRE is a new, freely-available cross-domain benchmark for Relation Extraction (RE)
We provide an empirical evaluation with a state-of-the-art model for relation classification.
As the meta-data enables us to shed new light on the state-of-the-art model, we provide a comprehensive analysis on the impact of difficult cases.
- Score: 21.513743126525622
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Relation Extraction (RE) has attracted increasing attention, but current RE
evaluation is limited to in-domain evaluation setups. Little is known on how
well a RE system fares in challenging, but realistic out-of-distribution
evaluation setups. To address this gap, we propose CrossRE, a new,
freely-available cross-domain benchmark for RE, which comprises six distinct
text domains and includes multi-label annotations. An additional innovation is
that we release meta-data collected during annotation, to include explanations
and flags of difficult instances. We provide an empirical evaluation with a
state-of-the-art model for relation classification. As the meta-data enables us
to shed new light on the state-of-the-art model, we provide a comprehensive
analysis on the impact of difficult cases and find correlations between model
and human annotations. Overall, our empirical investigation highlights the
difficulty of cross-domain RE. We release our dataset, to spur more research in
this direction.
Related papers
- Understanding the Cross-Domain Capabilities of Video-Based Few-Shot Action Recognition Models [3.072340427031969]
Few-shot action recognition (FSAR) aims to learn a model capable of identifying novel actions in videos using only a few examples.
In assuming the base dataset seen during meta-training and novel dataset used for evaluation can come from different domains, cross-domain few-shot learning alleviates data collection and annotation costs.
We systematically evaluate existing state-of-the-art single-domain, transfer-based, and cross-domain FSAR methods on new cross-domain tasks.
arXiv Detail & Related papers (2024-06-03T07:48:18Z) - Get Your Embedding Space in Order: Domain-Adaptive Regression for Forest Monitoring [1.4403877669472167]
We introduce a new dataset with aerial and satellite imagery in five countries with three forest-related regression tasks.
We compare methods through a restrictive setup where no prior on the target domain is available during training.
Building on the assumption that ordered relationships generalize better, we propose manifold diffusion for regression.
arXiv Detail & Related papers (2024-05-01T13:49:09Z) - Cross-Domain Few-Shot Segmentation via Iterative Support-Query
Correspondence Mining [81.09446228688559]
Cross-Domain Few-Shots (CD-FSS) poses the challenge of segmenting novel categories from a distinct domain using only limited exemplars.
We propose a novel cross-domain fine-tuning strategy that addresses the challenging CD-FSS tasks.
arXiv Detail & Related papers (2024-01-16T14:45:41Z) - Plot Retrieval as an Assessment of Abstract Semantic Association [131.58819293115124]
Text pairs in Plot Retrieval have less word overlap and more abstract semantic association.
Plot Retrieval can be the benchmark for further research on the semantic association modeling ability of IR models.
arXiv Detail & Related papers (2023-11-03T02:02:43Z) - A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers [76.51245425667845]
Relation extraction (RE) involves identifying the relations between entities from underlying content.
Deep neural networks have dominated the field of RE and made noticeable progress.
This survey is expected to facilitate researchers' collaborative efforts to address the challenges of real-world RE systems.
arXiv Detail & Related papers (2023-06-03T08:39:25Z) - Silver Syntax Pre-training for Cross-Domain Relation Extraction [20.603482820770356]
Relation Extraction (RE) remains a challenging task, especially when considering realistic out-of-domain evaluations.
obtaining high-quality (manually annotated) data is extremely expensive and cannot realistically be repeated for each new domain.
An intermediate training step on data from related tasks has shown to be beneficial across many NLP tasks.However, this setup still requires supplementary annotated data, which is often not available.
In this paper, we investigate intermediate pre-training specifically for RE. We exploit the affinity between syntactic structure and semantic RE, and identify the syntactic relations closely related to RE by being on the shortest dependency path between two entities
arXiv Detail & Related papers (2023-05-18T14:49:19Z) - Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual
Recognition [57.08108545219043]
Recognizing novel sub-categories with scarce samples is an essential and challenging research topic in computer vision.
Existing literature addresses this challenge by employing local-based representation approaches.
This article proposes a novel model, Robust Saliency-aware Distillation (RSaD), for few-shot fine-grained visual recognition.
arXiv Detail & Related papers (2023-05-12T00:13:17Z) - Should We Rely on Entity Mentions for Relation Extraction? Debiasing
Relation Extraction with Counterfactual Analysis [60.83756368501083]
We propose the CORE (Counterfactual Analysis based Relation Extraction) debiasing method for sentence-level relation extraction.
Our CORE method is model-agnostic to debias existing RE systems during inference without changing their training processes.
arXiv Detail & Related papers (2022-05-08T05:13:54Z) - What do You Mean by Relation Extraction? A Survey on Datasets and Study
on Scientific Relation Classification [21.513743126525622]
We present an empirical study on scientific Relation Classification across two datasets.
Despite large data overlap, our analysis reveals substantial discrepancies in annotation.
Variation within further sub-domains exists but impacts Relation Classification only limited degrees.
arXiv Detail & Related papers (2022-04-28T14:07:25Z) - Automatically Generating Counterfactuals for Relation Exaction [18.740447044960796]
relation extraction (RE) is a fundamental task in natural language processing.
Current deep neural models have achieved high accuracy but are easily affected by spurious correlations.
We develop a novel approach to derive contextual counterfactuals for entities.
arXiv Detail & Related papers (2022-02-22T04:46:10Z) - CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural
Summarization Systems [121.78477833009671]
We investigate the performance of different summarization models under a cross-dataset setting.
A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
arXiv Detail & Related papers (2020-10-11T02:19:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.