PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with
Relation-Aware Score Calibration
- URL: http://arxiv.org/abs/2309.13869v1
- Date: Mon, 25 Sep 2023 04:42:39 GMT
- Title: PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with
Relation-Aware Score Calibration
- Authors: Minseok Choi, Hyesu Lim, Jaegul Choo
- Abstract summary: Document-level relation extraction (DocRE) aims to extract relations of all entity pairs in a document.
Key challenge in DocRE is the cost of annotating such data which requires intensive human effort.
We propose PRiSM, which learns to adapt logits based on relation semantic information.
- Score: 44.074482478955126
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Document-level relation extraction (DocRE) aims to extract relations of all
entity pairs in a document. A key challenge in DocRE is the cost of annotating
such data which requires intensive human effort. Thus, we investigate the case
of DocRE in a low-resource setting, and we find that existing models trained on
low data overestimate the NA ("no relation") label, causing limited
performance. In this work, we approach the problem from a calibration
perspective and propose PRiSM, which learns to adapt logits based on relation
semantic information. We evaluate our method on three DocRE datasets and
demonstrate that integrating existing models with PRiSM improves performance by
as much as 26.38 F1 score, while the calibration error drops as much as 36
times when trained with about 3% of data. The code is publicly available at
https://github.com/brightjade/PRiSM.
Related papers
- Consistent Document-Level Relation Extraction via Counterfactuals [47.75615221596254]
It has been shown that document-level relation extraction models trained on real-world data suffer from factual biases.
We present CovEReD, a dataset of document-level counterfactual data for document extraction.
We show that by generating document-level counterfactual data with CovEReD models on them, consistency is maintained.
arXiv Detail & Related papers (2024-07-09T09:21:55Z) - Efficient Document Ranking with Learnable Late Interactions [73.41976017860006]
Cross-Encoder (CE) and Dual-Encoder (DE) models are two fundamental approaches for query-document relevance in information retrieval.
To predict relevance, CE models use joint query-document embeddings, while DE models maintain factorized query and document embeddings.
Recently, late-interaction models have been proposed to realize more favorable latency-quality tradeoffs, by using a DE structure followed by a lightweight scorer.
arXiv Detail & Related papers (2024-06-25T22:50:48Z) - Consistency Guided Knowledge Retrieval and Denoising in LLMs for
Zero-shot Document-level Relation Triplet Extraction [43.50683283748675]
Document-level Relation Triplet Extraction (DocRTE) is a fundamental task in information systems that aims to simultaneously extract entities with semantic relations from a document.
Existing methods heavily rely on a substantial amount of fully labeled data.
Recent advanced Large Language Models (LLMs), such as ChatGPT and LLaMA, exhibit impressive long-text generation capabilities.
arXiv Detail & Related papers (2024-01-24T17:04:28Z) - Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models [29.94694305204144]
We present a novel framework for document-level in-context few-shot relation extraction.
We evaluate our framework using DocRED, the largest publicly available dataset for document-level relation extraction.
arXiv Detail & Related papers (2023-10-17T09:10:27Z) - Does Recommend-Revise Produce Reliable Annotations? An Analysis on
Missing Instances in DocRED [60.39125850987604]
We show that a textit-revise scheme results in false negative samples and an obvious bias towards popular entities and relations.
The relabeled dataset is released to serve as a more reliable test set of document RE models.
arXiv Detail & Related papers (2022-04-17T11:29:01Z) - Document-Level Relation Extraction with Adaptive Focal Loss and
Knowledge Distillation [44.39395368597772]
Document-level Relation Extraction (DocRE) is a more challenging task compared to its sentence-level counterpart.
We propose a semi-supervised framework for DocRE with three novel components.
arXiv Detail & Related papers (2022-03-21T11:48:40Z) - SAIS: Supervising and Augmenting Intermediate Steps for Document-Level
Relation Extraction [51.27558374091491]
We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction.
Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
arXiv Detail & Related papers (2021-09-24T17:37:35Z) - Integrating Semantics and Neighborhood Information with Graph-Driven
Generative Models for Document Retrieval [51.823187647843945]
In this paper, we encode the neighborhood information with a graph-induced Gaussian distribution, and propose to integrate the two types of information with a graph-driven generative model.
Under the approximation, we prove that the training objective can be decomposed into terms involving only singleton or pairwise documents, enabling the model to be trained as efficiently as uncorrelated ones.
arXiv Detail & Related papers (2021-05-27T11:29:03Z) - Document-Level Relation Extraction with Adaptive Thresholding and
Localized Context Pooling [34.93480801598084]
One document commonly contains multiple entity pairs, and one entity pair occurs multiple times in the document associated with multiple possible relations.
We propose two novel techniques, adaptive thresholding and localized context pooling, to solve the multi-label and multi-entity problems.
arXiv Detail & Related papers (2020-10-21T20:41:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.