Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge

• URL: http://arxiv.org/abs/2010.13057v1
• Date: Sun, 25 Oct 2020 07:56:52 GMT
• Title: Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge
• Authors: Sathvik Nair, Mahesh Srinivasan, Stephan Meylan
• Abstract summary: We investigate whether recent advances in NLP, specifically contextualized word embeddings, capture human-like distinctions between English word senses. We find that participants' judgments of the relatedness between senses are correlated with distances between senses in the BERT embedding space. Our findings point towards the potential utility of continuous-space representations of sense meanings.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding context-dependent variation in word meanings is a key aspect of human language comprehension supported by the lexicon. Lexicographic resources (e.g., WordNet) capture only some of this context-dependent variation; for example, they often do not encode how closely senses, or discretized word meanings, are related to one another. Our work investigates whether recent advances in NLP, specifically contextualized word embeddings, capture human-like distinctions between English word senses, such as polysemy and homonymy. We collect data from a behavioral, web-based experiment, in which participants provide judgments of the relatedness of multiple WordNet senses of a word in a two-dimensional spatial arrangement task. We find that participants' judgments of the relatedness between senses are correlated with distances between senses in the BERT embedding space. Homonymous senses (e.g., bat as mammal vs. bat as sports equipment) are reliably more distant from one another in the embedding space than polysemous ones (e.g., chicken as animal vs. chicken as meat). Our findings point towards the potential utility of continuous-space representations of sense meanings.

Related papers

• Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models [53.840982361119565]
  The Composition Score is a novel model-based metric designed to quantify the degree of meaning composition during sentence comprehension. Experimental findings show that this metric correlates with brain clusters associated with word frequency, structural processing, and general sensitivity to words.
  arXiv  Detail & Related papers  (2024-03-07T08:44:42Z)
• Quantifying the redundancy between prosody and text [67.07817268372743]
  We use large language models to estimate how much information is redundant between prosody and the words themselves. We find a high degree of redundancy between the information carried by the words and prosodic information across several prosodic features. Still, we observe that prosodic features can not be fully predicted from text, suggesting that prosody carries information above and beyond the words.
  arXiv  Detail & Related papers  (2023-11-28T21:15:24Z)
• Can Word Sense Distribution Detect Semantic Changes of Words? [35.17635565325166]
  We show that word sense distributions can be accurately used to predict semantic changes of words in English, German, Swedish and Latin. Our experimental results on SemEval 2020 Task 1 dataset show that word sense distributions can be accurately used to predict semantic changes of words.
  arXiv  Detail & Related papers  (2023-10-16T13:41:27Z)
• Word sense extension [8.939269057094661]
  We present a paradigm of word sense extension (WSE) that enables words to spawn new senses toward novel context. We develop a framework that simulates novel word sense extension by partitioning a polysemous word type into two pseudo-tokens that mark its different senses. Our framework combines cognitive models of chaining with a learning scheme that transforms a language model embedding space to support various types of word sense extension.
  arXiv  Detail & Related papers  (2023-06-09T00:54:21Z)
• Neighboring Words Affect Human Interpretation of Saliency Explanations [65.29015910991261]
  Word-level saliency explanations are often used to communicate feature-attribution in text-based models. Recent studies found that superficial factors such as word length can distort human interpretation of the communicated saliency scores. We investigate how the marking of a word's neighboring words affect the explainee's perception of the word's importance in the context of a saliency explanation.
  arXiv  Detail & Related papers  (2023-05-04T09:50:25Z)
• Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT [64.40111510974957]
  We test whether meaning interferes with subject-verb number agreement in English. We generate semantically well-formed and nonsensical items. We find that BERT and humans are both sensitive to our semantic manipulation.
  arXiv  Detail & Related papers  (2022-09-21T17:57:23Z)
• Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings [11.475144702935568]
  We quantify how much the contextualized embeddings of each word sense vary across contexts in typical pre-trained models. We find that word representations are position-biased, where the first words in different contexts tend to be more similar.
  arXiv  Detail & Related papers  (2022-08-20T12:27:25Z)
• Exploring the Sensory Spaces of English Perceptual Verbs in Natural Language Data [0.40611352512781856]
  We focus on the most frequent perception verbs of English analyzed from an and Agentive vs. Experiential distinction. In this study we report on a data-driven approach based on distributional-semantic word embeddings and clustering models.
  arXiv  Detail & Related papers  (2021-10-19T03:58:44Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English) [2.792030485253753]
  We evaluate how well contextualized embeddings accommodate the continuous, dynamic nature of word meaning. We show that cosine distance systematically underestimates how similar humans find uses of the same sense of a word to be. We propose a synthesis between psycholinguistic theories of the mental lexicon and computational models of lexical semantics.
  arXiv  Detail & Related papers  (2021-05-27T16:07:13Z)
• Speakers Fill Lexical Semantic Gaps with Context [65.08205006886591]
  We operationalise the lexical ambiguity of a word as the entropy of meanings it can take. We find significant correlations between our estimate of ambiguity and the number of synonyms a word has in WordNet. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.
  arXiv  Detail & Related papers  (2020-10-05T17:19:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.