Related papers: Linguistic dependencies and statistical dependence

Linguistic dependencies and statistical dependence

URL: http://arxiv.org/abs/2104.08685v1
Date: Sun, 18 Apr 2021 02:43:37 GMT
Title: Linguistic dependencies and statistical dependence
Authors: Jacob Louis Hoover, Alessandro Sordoni, Wenyu Du, Timothy J. O'Donnell
Abstract summary: We use pretrained language models to estimate probabilities of words in context. We find that maximum-CPMI trees correspond to linguistic dependencies more often than trees extracted from non-contextual PMI estimate.
Score: 76.89273585568084
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: What is the relationship between linguistic dependencies and statistical dependence? Building on earlier work in NLP and cognitive science, we study this question. We introduce a contextualized version of pointwise mutual information (CPMI), using pretrained language models to estimate probabilities of words in context. Extracting dependency trees which maximize CPMI, we compare the resulting structures against gold dependencies. Overall, we find that these maximum-CPMI trees correspond to linguistic dependencies more often than trees extracted from non-contextual PMI estimate, but only roughly as often as a simple baseline formed by connecting adjacent words. We also provide evidence that the extent to which the two kinds of dependency align cannot be explained by the distance between words or by the category of the dependency relation. Finally, our analysis sheds some light on the differences between large pretrained language models, specifically in the kinds of inductive biases they encode.

Related papers

A Novel Dependency Framework for Enhancing Discourse Data Analysis [27.152245569974678]
This study has as its primary focus the conversion of PDTB annotations into dependency structures. It employs refined BERT-based discourses to test the validity of the dependency data derived from the PDTB-style corpora in English, Chinese, and several other languages. The results show that the PDTB dependency data is valid and that there is a strong correlation between the two types of dependency distance.
arXiv Detail & Related papers (2024-07-17T10:55:00Z)
Context versus Prior Knowledge in Language Models [49.17879668110546]
Language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We propose two mutual information-based metrics to measure a model's dependency on a context and on its prior about an entity.
arXiv Detail & Related papers (2024-04-06T13:46:53Z)
Exploring language relations through syntactic distances and geographic proximity [0.4369550829556578]
We explore linguistic distances using series of parts of speech (POS) extracted from the Universal Dependencies dataset. We find definite clusters that correspond to well known language families and groups, with exceptions explained by distinct morphological typologies.
arXiv Detail & Related papers (2024-03-27T10:36:17Z)
Syntactic Language Change in English and German: Metrics, Parsers, and Convergences [56.47832275431858]
The current paper looks at diachronic trends in syntactic language change in both English and German, using corpora of parliamentary debates from the last c. 160 years. We base our observations on five dependencys, including the widely used Stanford Core as well as 4 newer alternatives. We show that changes in syntactic measures seem to be more frequent at the tails of sentence length distributions.
arXiv Detail & Related papers (2024-02-18T11:46:16Z)
To be Closer: Learning to Link up Aspects with Opinions [18.956990787407793]
Dependency parse trees are helpful for discovering the opinion words in aspect-based sentiment analysis (ABSA) In this work, we aim to shorten the distance between aspects and corresponding opinion words by learning an aspect-centric tree structure. The learning process allows the tree structure to adaptively correlate the aspect and opinion words, enabling us to better identify the polarity in the ABSA task.
arXiv Detail & Related papers (2021-09-17T07:37:13Z)
Learning compositional structures for semantic graph parsing [81.41592892863979]
We show how AM dependency parsing can be trained directly on a neural latent-variable model. Our model picks up on several linguistic phenomena on its own and achieves comparable accuracy to supervised training.
arXiv Detail & Related papers (2021-06-08T14:20:07Z)
Syntactic Nuclei in Dependency Parsing -- A Multilingual Exploration [8.25332300240617]
We show how the concept of nucleus can be defined in the framework of Universal Dependencies. Experiments on 12 languages show that nucleus composition gives small but significant improvements in parsing accuracy.
arXiv Detail & Related papers (2021-01-28T12:22:30Z)
Learning Relation Prototype from Unlabeled Texts for Long-tail Relation Extraction [84.64435075778988]
We propose a general approach to learn relation prototypes from unlabeled texts. We learn relation prototypes as an implicit factor between entities. We conduct experiments on two publicly available datasets: New York Times and Google Distant Supervision.
arXiv Detail & Related papers (2020-11-27T06:21:12Z)
Learning Informative Representations of Biomedical Relations with Latent Variable Models [2.4366811507669115]
We propose a latent variable model with an arbitrarily flexible distribution to represent the relation between an entity pair. We demonstrate that our model achieves results competitive with strong baselines for both tasks while having fewer parameters and being significantly faster to train.
arXiv Detail & Related papers (2020-11-20T08:56:31Z)
Please Mind the Root: Decoding Arborescences for Dependency Parsing [67.71280539312536]
We analyze the output of state-of-the-arts on many languages from the Universal Dependency Treebank. The worst constraint-violation rate we observe is 24%.
arXiv Detail & Related papers (2020-10-06T08:31:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.