Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings

• URL: http://arxiv.org/abs/2208.09669v1
• Date: Sat, 20 Aug 2022 12:27:25 GMT
• Title: Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings
• Authors: Yile Wang and Yue Zhang
• Abstract summary: We quantify how much the contextualized embeddings of each word sense vary across contexts in typical pre-trained models. We find that word representations are position-biased, where the first words in different contexts tend to be more similar.
• Score: 11.475144702935568
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contextualized word embeddings in language models have given much advance to NLP. Intuitively, sentential information is integrated into the representation of words, which can help model polysemy. However, context sensitivity also leads to the variance of representations, which may break the semantic consistency for synonyms. We quantify how much the contextualized embeddings of each word sense vary across contexts in typical pre-trained models. Results show that contextualized embeddings can be highly consistent across contexts. In addition, part-of-speech, number of word senses, and sentence length have an influence on the variance of sense representations. Interestingly, we find that word representations are position-biased, where the first words in different contexts tend to be more similar. We analyze such a phenomenon and also propose a simple way to alleviate such bias in distance-based word sense disambiguation settings.

Related papers

• Investigating Idiomaticity in Word Representations [9.208145117062339]
  We focus on noun compounds of varying levels of idiomaticity in two languages (English and Portuguese) We present a dataset of minimal pairs containing human idiomaticity judgments for each noun compound at both type and token levels. We define a set of fine-grained metrics of Affinity and Scaled Similarity to determine how sensitive the models are to perturbations that may lead to changes in idiomaticity.
  arXiv  Detail & Related papers  (2024-11-04T21:05:01Z)
• Conjuring Semantic Similarity [59.18714889874088]
  The semantic similarity between two textual expressions measures the distance between their latent'meaning' We propose a novel approach whereby the semantic similarity among textual expressions is based not on other expressions they can be rephrased as, but rather based on the imagery they evoke. Our method contributes a novel perspective on semantic similarity that not only aligns with human-annotated scores, but also opens up new avenues for the evaluation of text-conditioned generative models.
  arXiv  Detail & Related papers  (2024-10-21T18:51:34Z)
• The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations [3.4668147567693453]
  The quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words. Our analysis reveals, among other interesting findings, that the quality of representations of words that are split is often, but not always, worse than that of the embeddings of known words.
  arXiv  Detail & Related papers  (2024-02-22T15:04:24Z)
• Neighboring Words Affect Human Interpretation of Saliency Explanations [65.29015910991261]
  Word-level saliency explanations are often used to communicate feature-attribution in text-based models. Recent studies found that superficial factors such as word length can distort human interpretation of the communicated saliency scores. We investigate how the marking of a word's neighboring words affect the explainee's perception of the word's importance in the context of a saliency explanation.
  arXiv  Detail & Related papers  (2023-05-04T09:50:25Z)
• SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings [4.479834103607384]
  Adding interpretability to word embeddings represents an area of active research in text representation. We present SensePOLAR, an extension of the original POLAR framework that enables word-sense aware interpretability for pre-trained contextual word embeddings.
  arXiv  Detail & Related papers  (2023-01-11T20:25:53Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy [0.0]
  We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations. Experiments are performed in Galician, Portuguese, English, and Spanish.
  arXiv  Detail & Related papers  (2021-06-25T10:54:23Z)
• Understanding Synonymous Referring Expressions via Contrastive Features [105.36814858748285]
  We develop an end-to-end trainable framework to learn contrastive features on the image and object instance levels. We conduct extensive experiments to evaluate the proposed algorithm on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-20T17:56:24Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Speakers Fill Lexical Semantic Gaps with Context [65.08205006886591]
  We operationalise the lexical ambiguity of a word as the entropy of meanings it can take. We find significant correlations between our estimate of ambiguity and the number of synonyms a word has in WordNet. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.
  arXiv  Detail & Related papers  (2020-10-05T17:19:10Z)
• Comparative Analysis of Word Embeddings for Capturing Word Similarities [0.0]
  Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks. Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embeddings. selecting the appropriate word embeddings is a perplexing task since the projected embedding space is not intuitive to humans.
  arXiv  Detail & Related papers  (2020-05-08T01:16:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.