GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction

• URL: http://arxiv.org/abs/2010.03009v2
• Date: Wed, 17 Feb 2021 20:11:41 GMT
• Title: GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction
• Authors: Wasi Uddin Ahmad and Nanyun Peng and Kai-Wei Chang
• Abstract summary: We introduce graph convolutional networks (GCNs) with universal dependency parses to learn language-agnostic sentence representations. GCNs struggle to model words with long-range dependencies or are not directly connected in the dependency tree. We propose to utilize the self-attention mechanism to learn the dependencies between words with different syntactic distances.
• Score: 107.8262586956778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in cross-lingual relation and event extraction use graph convolutional networks (GCNs) with universal dependency parses to learn language-agnostic sentence representations such that models trained on one language can be applied to other languages. However, GCNs struggle to model words with long-range dependencies or are not directly connected in the dependency tree. To address these challenges, we propose to utilize the self-attention mechanism where we explicitly fuse structural information to learn the dependencies between words with different syntactic distances. We introduce GATE, a {\bf G}raph {\bf A}ttention {\bf T}ransformer {\bf E}ncoder, and test its cross-lingual transferability on relation and event extraction tasks. We perform experiments on the ACE05 dataset that includes three typologically different languages: English, Chinese, and Arabic. The evaluation results show that GATE outperforms three recently proposed methods by a large margin. Our detailed analysis reveals that due to the reliance on syntactic dependencies, GATE produces robust representations that facilitate transfer across languages.

Related papers

• mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning [54.523172171533645]
  Cross-lingual named entity recognition (CrossNER) faces challenges stemming from uneven performance due to the scarcity of multilingual corpora. We propose Multi-view Contrastive Learning for Cross-lingual Named Entity Recognition (mCL-NER) Our experiments on the XTREME benchmark, spanning 40 languages, demonstrate the superiority of mCL-NER over prior data-driven and model-based approaches.
  arXiv  Detail & Related papers  (2023-08-17T16:02:29Z)
• Transferring Knowledge Distillation for Multilingual Social Event Detection [42.663309895263666]
  Recently published graph neural networks (GNNs) show promising performance at social event detection tasks. We present a GNN that incorporates cross-lingual word embeddings for detecting events in multilingual data streams. Experiments on both synthetic and real-world datasets show the framework to be highly effective at detection in both multilingual data and in languages where training samples are scarce.
  arXiv  Detail & Related papers  (2021-08-06T12:38:42Z)
• Cross-Lingual Adaptation for Type Inference [29.234418962960905]
  We propose a cross-lingual adaptation framework, PLATO, to transfer a deep learning-based type inference procedure across weakly typed languages. By leveraging data from strongly typed languages, PLATO improves the perplexity of the backbone cross-programming-language model.
  arXiv  Detail & Related papers  (2021-07-01T00:20:24Z)
• Cross-lingual Text Classification with Heterogeneous Graph Neural Network [2.6936806968297913]
  Cross-lingual text classification aims at training a classifier on the source language and transferring the knowledge to target languages. Recent multilingual pretrained language models (mPLM) achieve impressive results in cross-lingual classification tasks. We propose a simple yet effective method to incorporate heterogeneous information within and across languages for cross-lingual text classification.
  arXiv  Detail & Related papers  (2021-05-24T12:45:42Z)
• Multilingual Autoregressive Entity Linking [49.35994386221958]
  mGENRE is a sequence-to-sequence system for the Multilingual Entity Linking problem. For a mention in a given language, mGENRE predicts the name of the target entity left-to-right, token-by-token. We show the efficacy of our approach through extensive evaluation including experiments on three popular MEL benchmarks.
  arXiv  Detail & Related papers  (2021-03-23T13:25:55Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• Language and Visual Entity Relationship Graph for Agent Navigation [54.059606864535304]
  Vision-and-Language Navigation (VLN) requires an agent to navigate in a real-world environment following natural language instructions. We propose a novel Language and Visual Entity Relationship Graph for modelling the inter-modal relationships between text and vision. Experiments show that by taking advantage of the relationships we are able to improve over state-of-the-art.
  arXiv  Detail & Related papers  (2020-10-19T08:25:55Z)
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision [110.66085917826648]
  We develop a technique that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images. "vokenization" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks.
  arXiv  Detail & Related papers  (2020-10-14T02:11:51Z)
• Cross-Lingual Adaptation Using Universal Dependencies [1.027974860479791]
  We show that models trained using UD parse trees for complex NLP tasks can characterize very different languages. Based on UD parse trees, we develop several models using tree kernels and show that these models trained on the English dataset can correctly classify data of other languages.
  arXiv  Detail & Related papers  (2020-03-24T13:04:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.