Universal Dependencies according to BERT: both more specific and more general

• URL: http://arxiv.org/abs/2004.14620v3
• Date: Tue, 6 Oct 2020 10:22:33 GMT
• Title: Universal Dependencies according to BERT: both more specific and more general
• Authors: Tomasz Limisiewicz and Rudolf Rosa and David Mare\v{c}ek
• Abstract summary: This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT.
• Score: 4.63257209402195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.

Related papers

• Entity-Aware Self-Attention and Contextualized GCN for Enhanced Relation Extraction in Long Sentences [5.453850739960517]
  We propose a novel model, Entity-aware Self-attention Contextualized GCN (ESC-GCN), which efficiently incorporates syntactic structure of input sentences and semantic context of sequences. Our model achieves encouraging performance as compared to existing dependency-based and sequence-based models.
  arXiv  Detail & Related papers  (2024-09-15T10:50:51Z)
• Entity Disambiguation via Fusion Entity Decoding [68.77265315142296]
  We propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. We observe +1.5% improvements in end-to-end entity linking in the GERBIL benchmark compared with EntQA.
  arXiv  Detail & Related papers  (2024-04-02T04:27:54Z)
• Data Augmentation for Machine Translation via Dependency Subtree Swapping [0.0]
  We present a generic framework for data augmentation via dependency subtree swapping. We extract corresponding subtrees from the dependency parse trees of the source and target sentences and swap these across bisentences to create augmented samples. We conduct resource-constrained experiments on 4 language pairs in both directions using the IWSLT text translation datasets and the Hunglish2 corpus.
  arXiv  Detail & Related papers  (2023-07-13T19:00:26Z)
• Relation-Specific Attentions over Entity Mentions for Enhanced Document-Level Relation Extraction [4.685620089585031]
  We propose RSMAN which performs selective attentions over different entity mentions with respect to candidate relations. Our experiments upon two benchmark datasets show that our RSMAN can bring significant improvements for some backbone models.
  arXiv  Detail & Related papers  (2022-05-28T10:40:31Z)
• TAGPRIME: A Unified Framework for Relational Structure Extraction [71.88926365652034]
  TAGPRIME is a sequence tagging model that appends priming words about the information of the given condition to the input text. With the self-attention mechanism in pre-trained language models, the priming words make the output contextualized representations contain more information about the given condition. Extensive experiments and analyses on three different tasks that cover ten datasets across five different languages demonstrate the generality and effectiveness of TAGPRIME.
  arXiv  Detail & Related papers  (2022-05-25T08:57:46Z)
• Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task [70.29624135819884]
  We study the extent to which BERT is able to perform lexically-independent subject-verb number agreement (NA) on targeted syntactic templates. Our results on nonce sentences suggest that the model generalizes well for simple templates, but fails to perform lexically-independent syntactic generalization when as little as one attractor is present.
  arXiv  Detail & Related papers  (2022-04-14T11:33:15Z)
• Entity Relation Extraction as Dependency Parsing in Visually Rich Documents [18.67730663266417]
  We adapt the popular dependency parsing model, the biaffine, to this entity relation extraction task. Being different from the original dependency parsing model which recognizes dependency relations between words, we identify relations between groups of words with layout information instead. As for the real-world application, our model has been applied to the in-house customs data, achieving reliable performance in the production setting.
  arXiv  Detail & Related papers  (2021-10-19T12:26:40Z)
• Linguistic dependencies and statistical dependence [76.89273585568084]
  We use pretrained language models to estimate probabilities of words in context. We find that maximum-CPMI trees correspond to linguistic dependencies more often than trees extracted from non-contextual PMI estimate.
  arXiv  Detail & Related papers  (2021-04-18T02:43:37Z)
• Improving Distantly-Supervised Relation Extraction through BERT-based Label & Instance Embeddings [2.88848244747161]
  We propose REDSandT, a novel distantly-supervised transformer-based RE method. We exploit BERT's pre-trained model, and the relationship between labels and entities, respectively. Experiments in the NYT-10 dataset show that REDSandT captures a broader set of relations with higher confidence.
  arXiv  Detail & Related papers  (2021-02-01T20:50:24Z)
• Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction [84.64435075778988]
  We propose a general approach to learn relation prototypes from unlabeled texts. We learn relation prototypes as an implicit factor between entities. We conduct experiments on two publicly available datasets: New York Times and Google Distant Supervision.
  arXiv  Detail & Related papers  (2020-11-27T06:21:12Z)
• Please Mind the Root: Decoding Arborescences for Dependency Parsing [67.71280539312536]
  We analyze the output of state-of-the-arts on many languages from the Universal Dependency Treebank. The worst constraint-violation rate we observe is 24%.
  arXiv  Detail & Related papers  (2020-10-06T08:31:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.