Patterns of Lexical Ambiguity in Contextualised Language Models
- URL: http://arxiv.org/abs/2109.13032v2
- Date: Wed, 29 Sep 2021 12:40:45 GMT
- Title: Patterns of Lexical Ambiguity in Contextualised Language Models
- Authors: Janosch Haber, Massimo Poesio
- Abstract summary: We introduce an extended, human-annotated dataset of graded word sense similarity and co-predication.
Both types of human judgements indicate that the similarity of polysemic interpretations falls in a continuum between identity of meaning and homonymy.
Our dataset appears to capture a substantial part of the complexity of lexical ambiguity, and can provide a realistic test bed for contextualised embeddings.
- Score: 9.747449805791092
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: One of the central aspects of contextualised language models is that they
should be able to distinguish the meaning of lexically ambiguous words by their
contexts. In this paper we investigate the extent to which the contextualised
embeddings of word forms that display multiplicity of sense reflect traditional
distinctions of polysemy and homonymy. To this end, we introduce an extended,
human-annotated dataset of graded word sense similarity and co-predication
acceptability, and evaluate how well the similarity of embeddings predicts
similarity in meaning. Both types of human judgements indicate that the
similarity of polysemic interpretations falls in a continuum between identity
of meaning and homonymy. However, we also observe significant differences
within the similarity ratings of polysemes, forming consistent patterns for
different types of polysemic sense alternation. Our dataset thus appears to
capture a substantial part of the complexity of lexical ambiguity, and can
provide a realistic test bed for contextualised embeddings. Among the tested
models, BERT Large shows the strongest correlation with the collected word
sense similarity ratings, but struggles to consistently replicate the observed
similarity patterns. When clustering ambiguous word forms based on their
embeddings, the model displays high confidence in discerning homonyms and some
types of polysemic alternations, but consistently fails for others.
Related papers
- Interpretable Measures of Conceptual Similarity by
Complexity-Constrained Descriptive Auto-Encoding [112.0878081944858]
Quantifying the degree of similarity between images is a key copyright issue for image-based machine learning.
We seek to define and compute a notion of "conceptual similarity" among images that captures high-level relations.
Two highly dissimilar images can be discriminated early in their description, whereas conceptually dissimilar ones will need more detail to be distinguished.
arXiv Detail & Related papers (2024-02-14T03:31:17Z) - Agentivit\`a e telicit\`a in GilBERTo: implicazioni cognitive [77.71680953280436]
The goal of this study is to investigate whether a Transformer-based neural language model infers lexical semantics.
The semantic properties considered are telicity (also combined with definiteness) and agentivity.
arXiv Detail & Related papers (2023-07-06T10:52:22Z) - Universality and diversity in word patterns [0.0]
We present an analysis of lexical statistical connections for eleven major languages.
We find that the diverse manners that languages utilize to express word relations give rise to unique pattern distributions.
arXiv Detail & Related papers (2022-08-23T20:03:27Z) - Lost in Context? On the Sense-wise Variance of Contextualized Word
Embeddings [11.475144702935568]
We quantify how much the contextualized embeddings of each word sense vary across contexts in typical pre-trained models.
We find that word representations are position-biased, where the first words in different contexts tend to be more similar.
arXiv Detail & Related papers (2022-08-20T12:27:25Z) - Keywords and Instances: A Hierarchical Contrastive Learning Framework
Unifying Hybrid Granularities for Text Generation [59.01297461453444]
We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text.
Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
arXiv Detail & Related papers (2022-05-26T13:26:03Z) - Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation.
This paper aims to address the issue with a mask-and-predict strategy.
We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions.
Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
arXiv Detail & Related papers (2021-10-04T03:59:15Z) - Exploring the Representation of Word Meanings in Context: A Case Study
on Homonymy and Synonymy [0.0]
We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations.
Experiments are performed in Galician, Portuguese, English, and Spanish.
arXiv Detail & Related papers (2021-06-25T10:54:23Z) - Topology of Word Embeddings: Singularities Reflect Polysemy [68.8204255655161]
We introduce a topological measure of polysemy based on persistent homology that correlates well with the actual number of meanings of a word.
We propose a simple, topologically motivated solution to the SemEval-2010 task on Word Sense Induction & Disambiguation.
arXiv Detail & Related papers (2020-11-18T17:21:51Z) - Bio-inspired Structure Identification in Language Embeddings [3.5292026405502215]
We present a series of explorations using bio-inspired methodology to traverse and visualize word embeddings.
We show that our model can be used to investigate how different word embedding techniques result in different semantic outputs.
arXiv Detail & Related papers (2020-09-05T04:44:15Z) - Comparative Analysis of Word Embeddings for Capturing Word Similarities [0.0]
Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks.
Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embeddings.
selecting the appropriate word embeddings is a perplexing task since the projected embedding space is not intuitive to humans.
arXiv Detail & Related papers (2020-05-08T01:16:03Z) - Multiplex Word Embeddings for Selectional Preference Acquisition [70.33531759861111]
We propose a multiplex word embedding model, which can be easily extended according to various relations among words.
Our model can effectively distinguish words with respect to different relations without introducing unnecessary sparseness.
arXiv Detail & Related papers (2020-01-09T04:47:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.