How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction

• URL: http://arxiv.org/abs/2107.12064v1
• Date: Mon, 26 Jul 2021 09:38:28 GMT
• Title: How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction
• Authors: Zikun Hu, Yixin Cao, Lifu Huang, Tat-Seng Chua
• Abstract summary: We quantitatively evaluate the effect of attention and Knowledge Graph on bag-level relation extraction (RE) We find that (1) higher attention accuracy may lead to worse performance as it may harm the model's ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns; and (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior.
• Score: 66.09605613944201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge Graph (KG) and attention mechanism have been demonstrated effective in introducing and selecting useful information for weakly supervised methods. However, only qualitative analysis and ablation study are provided as evidence. In this paper, we contribute a dataset and propose a paradigm to quantitatively evaluate the effect of attention and KG on bag-level relation extraction (RE). We find that (1) higher attention accuracy may lead to worse performance as it may harm the model's ability to extract entity mention features; (2) the performance of attention is largely influenced by various noise distribution patterns, which is closely related to real-world datasets; (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior; and (4) attention mechanism may exacerbate the issue of insufficient training data. Based on these findings, we show that a straightforward variant of RE model can achieve significant improvements (6% AUC on average) on two real-world datasets as compared with three state-of-the-art baselines. Our codes and datasets are available at https://github.com/zig-kwin-hu/how-KG-ATT-help.

Related papers

• Characterizing Massive Activations of Attention Mechanism in Graph Neural Networks [0.9499648210774584]
  Recently, attention mechanisms have been integrated into Graph Neural Networks (GNNs) to improve their ability to capture complex patterns. This paper presents the first comprehensive study revealing the emergence of Massive Activations (MAs) within attention layers. Our study assesses various GNN models using benchmark datasets, including ZINC, TOX21, and PROTEINS.
  arXiv  Detail & Related papers  (2024-09-05T12:19:07Z)
• Distilling Aggregated Knowledge for Weakly-Supervised Video Anomaly Detection [11.250490586786878]
  Video anomaly detection aims to develop automated models capable of identifying abnormal events in surveillance videos. We show that distilling knowledge from aggregated representations into a relatively simple model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-06-05T00:44:42Z)
• Data-Centric Long-Tailed Image Recognition [49.90107582624604]
  Long-tail models exhibit a strong demand for high-quality data. Data-centric approaches aim to enhance both the quantity and quality of data to improve model performance. There is currently a lack of research into the underlying mechanisms explaining the effectiveness of information augmentation.
  arXiv  Detail & Related papers  (2023-11-03T06:34:37Z)
• SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning [41.92038829041499]
  Multimodal Review Helpfulness Prediction (MRHP) aims to sort product reviews according to the predicted helpfulness scores. Previous work on this task focuses on attention-based modality fusion, information integration, and relation modeling. We propose SANCL: Selective Attention and Natural Contrastive Learning for MRHP.
  arXiv  Detail & Related papers  (2022-09-12T06:31:13Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• More Than Just Attention: Learning Cross-Modal Attentions with Contrastive Constraints [63.08768589044052]
  We propose Contrastive Content Re-sourcing ( CCR) and Contrastive Content Swapping ( CCS) constraints to address such limitation. CCR and CCS constraints supervise the training of attention models in a contrastive learning manner without requiring explicit attention annotations. Experiments on both Flickr30k and MS-COCO datasets demonstrate that integrating these attention constraints into two state-of-the-art attention-based models improves the model performance.
  arXiv  Detail & Related papers  (2021-05-20T08:48:10Z)
• Incorporating Causal Graphical Prior Knowledge into Predictive Modeling via Simple Data Augmentation [92.96204497841032]
  Causal graphs (CGs) are compact representations of the knowledge of the data generating processes behind the data distributions. We propose a model-agnostic data augmentation method that allows us to exploit the prior knowledge of the conditional independence (CI) relations. We experimentally show that the proposed method is effective in improving the prediction accuracy, especially in the small-data regime.
  arXiv  Detail & Related papers  (2021-02-27T06:13:59Z)
• Few-Shot Relation Learning with Attention for EEG-based Motor Imagery Classification [11.873435088539459]
  Brain-Computer Interfaces (BCI) based on Electroencephalography (EEG) signals have received a lot of attention. Motor imagery (MI) data can be used to aid rehabilitation as well as in autonomous driving scenarios. classification of MI signals is vital for EEG-based BCI systems.
  arXiv  Detail & Related papers  (2020-03-03T02:34:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.