Statistically significant detection of semantic shifts using contextual word embeddings

• URL: http://arxiv.org/abs/2104.03776v1
• Date: Thu, 8 Apr 2021 13:58:54 GMT
• Title: Statistically significant detection of semantic shifts using contextual word embeddings
• Authors: Yang Liu, Alan Medlar and Dorota Glowacka
• Abstract summary: We propose an approach to estimate semantic shifts by combining contextual word embeddings with permutation-based statistical tests. We demonstrate the performance of this approach in simulation, achieving consistently high precision by suppressing false positives. We additionally analyzed real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus.
• Score: 7.439525715543974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting lexical semantic shifts in smaller data sets, e.g. in historical linguistics and digital humanities, is challenging due to a lack of statistical power. This issue is exacerbated by non-contextual word embeddings that produce one embedding per token and therefore mask the variability present in the data. In this article, we propose an approach to estimate semantic shifts by combining contextual word embeddings with permutation-based statistical tests. Multiple comparisons are addressed using a false discovery rate procedure. We demonstrate the performance of this approach in simulation, achieving consistently high precision by suppressing false positives. We additionally analyzed real-world data from SemEval-2020 Task 1 and the Liverpool FC subreddit corpus. We show that by taking sample variation into account, we can improve the robustness of individual semantic shift estimates without degrading overall performance.

Related papers

• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• Statistical Uncertainty in Word Embeddings: GloVe-V [35.04183792123882]
  We introduce a method to obtain approximate, easy-to-use, and scalable reconstruction error variance estimates for GloVe. To demonstrate the value of embeddings with variance (GloVe-V), we illustrate how our approach enables principled hypothesis testing in core word embedding tasks.
  arXiv  Detail & Related papers  (2024-06-18T00:35:02Z)
• Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings [17.803726860514193]
  Detection of semantic variation of words is an important task for various NLP applications. We argue that mean representations alone cannot accurately capture such semantic variations. We propose a method that uses the entire cohort of the contextualised embeddings of the target word.
  arXiv  Detail & Related papers  (2023-05-15T13:58:21Z)
• TransDrift: Modeling Word-Embedding Drift using Transformer [8.707217592903735]
  We propose TransDrift, a transformer-based prediction model for word embeddings. Our model accurately learns the dynamics of the embedding drift and predicts the future embedding. Our embeddings lead to superior performance compared to the previous methods.
  arXiv  Detail & Related papers  (2022-06-16T10:48:26Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Semantic-Preserving Adversarial Text Attacks [85.32186121859321]
  We propose a Bigram and Unigram based adaptive Semantic Preservation Optimization (BU-SPO) method to examine the vulnerability of deep models. Our method achieves the highest attack success rates and semantics rates by changing the smallest number of words compared with existing methods.
  arXiv  Detail & Related papers  (2021-08-23T09:05:18Z)
• Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluation [109.06060143938052]
  We propose a "double perturbation" framework to uncover model weaknesses beyond the test dataset. We apply this framework to study two perturbation-based approaches that are used to analyze models' robustness and counterfactual bias in English.
  arXiv  Detail & Related papers  (2021-04-12T06:57:36Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• On the Sentence Embeddings from Pre-trained Language Models [78.45172445684126]
  In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. We propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective.
  arXiv  Detail & Related papers  (2020-11-02T13:14:57Z)
• Word Embeddings: Stability and Semantic Change [0.0]
  We present an experimental study on the instability of the training process of three of the most influential embedding techniques of the last decade: word2vec, GloVe and fastText. We propose a statistical model to describe the instability of embedding techniques and introduce a novel metric to measure the instability of the representation of an individual word.
  arXiv  Detail & Related papers  (2020-07-23T16:03:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.