Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling

• URL: http://arxiv.org/abs/2104.07704v2
• Date: Fri, 2 Jun 2023 09:06:59 GMT
• Title: Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labelling
• Authors: Alireza Mohammadshahi, James Henderson
• Abstract summary: We propose a Syntax-aware Graph-to-Graph Transformer (SynG2G-Tr) model, which encodes the syntactic structure using a novel way to input graph relations as embeddings. This approach adds a soft bias towards attention patterns that follow the syntactic structure but also allows the model to use this information to learn alternative patterns. We evaluate our model on both span-based and dependency-based SRL datasets, and outperform previous alternative methods in both in-domain and out-of-domain settings.
• Score: 18.028902306143102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent models have shown that incorporating syntactic knowledge into the semantic role labelling (SRL) task leads to a significant improvement. In this paper, we propose Syntax-aware Graph-to-Graph Transformer (SynG2G-Tr) model, which encodes the syntactic structure using a novel way to input graph relations as embeddings, directly into the self-attention mechanism of Transformer. This approach adds a soft bias towards attention patterns that follow the syntactic structure but also allows the model to use this information to learn alternative patterns. We evaluate our model on both span-based and dependency-based SRL datasets, and outperform previous alternative methods in both in-domain and out-of-domain settings, on CoNLL 2005 and CoNLL 2009 datasets.

Related papers

• Language Models are Graph Learners [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, including Graph Neural Networks (GNNs) and Graph Transformers (GTs) We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction [4.194768796374315]
  We propose a novel method for joint entity and relation extraction from unstructured text by framing it as a conditional sequence generation problem. It generates a linearized graph where nodes represent text spans and edges represent relation triplets. Our method employs a transformer encoder-decoder architecture with pointing mechanism on a dynamic vocabulary of spans and relation types.
  arXiv  Detail & Related papers  (2024-01-02T18:32:14Z)
• Incorporating Graph Information in Transformer-based AMR Parsing [34.461828101932184]
  LeakDistill is a model and method that explores a modification to the Transformer architecture. We show how, by employing word-to-node alignment to embed graph structural information into the encoder at training time, we can obtain state-of-the-art AMR parsing.
  arXiv  Detail & Related papers  (2023-06-23T12:12:08Z)
• Graph Adaptive Semantic Transfer for Cross-domain Sentiment Classification [68.06496970320595]
  Cross-domain sentiment classification (CDSC) aims to use the transferable semantics learned from the source domain to predict the sentiment of reviews in the unlabeled target domain. We present Graph Adaptive Semantic Transfer (GAST) model, an adaptive syntactic graph embedding method that is able to learn domain-invariant semantics from both word sequences and syntactic graphs.
  arXiv  Detail & Related papers  (2022-05-18T07:47:01Z)
• Hybrid Routing Transformer for Zero-Shot Learning [83.64532548391]
  This paper presents a novel transformer encoder-decoder model, called hybrid routing transformer (HRT) We embed an active attention, which is constructed by both the bottom-up and the top-down dynamic routing pathways to generate the attribute-aligned visual feature. While in HRT decoder, we use static routing to calculate the correlation among the attribute-aligned visual features, the corresponding attribute semantics, and the class attribute vectors to generate the final class label predictions.
  arXiv  Detail & Related papers  (2022-03-29T07:55:08Z)
• Structure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing [20.67024416678313]
  We explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We show that the proposed parsing architecture retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new state of the art for AMR 2.0, without the need for graph re-categorization.
  arXiv  Detail & Related papers  (2021-10-29T04:36:31Z)
• TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning [87.38675639186405]
  We propose a novel graph neural network approach, called TCL, which deals with the dynamically-evolving graph in a continuous-time fashion. To the best of our knowledge, this is the first attempt to apply contrastive learning to representation learning on dynamic graphs.
  arXiv  Detail & Related papers  (2021-05-17T15:33:25Z)
• R2D2: Relational Text Decoding with Transformers [18.137828323277347]
  We propose a novel framework for modeling the interaction between graphical structures and the natural language text associated with their nodes and edges. Our proposed method utilizes both the graphical structure as well as the sequential nature of the texts. While the proposed model has wide applications, we demonstrate its capabilities on data-to-text generation tasks.
  arXiv  Detail & Related papers  (2021-05-10T19:59:11Z)
• GTN-ED: Event Detection Using Graph Transformer Networks [12.96137943176861]
  We propose a novel framework for incorporating both dependencies and their labels using a recently proposed technique called Graph Transformer Networks (GTN) We integrate GTNs to leverage dependency relations on two existing homogeneous-graph-based models, and demonstrate an improvement in the F1 score on the ACE dataset.
  arXiv  Detail & Related papers  (2021-04-30T16:35:29Z)
• Lightweight, Dynamic Graph Convolutional Networks for AMR-to-Text Generation [56.73834525802723]
  Lightweight Dynamic Graph Convolutional Networks (LDGCNs) are proposed. LDGCNs capture richer non-local interactions by synthesizing higher order information from the input graphs. We develop two novel parameter saving strategies based on the group graph convolutions and weight tied convolutions to reduce memory usage and model complexity.
  arXiv  Detail & Related papers  (2020-10-09T06:03:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.