$\textit{Swap and Predict}$ -- Predicting the Semantic Changes in Words across Corpora by Context Swapping

• URL: http://arxiv.org/abs/2310.10397v1
• Date: Mon, 16 Oct 2023 13:39:44 GMT
• Title: $\textit{Swap and Predict}$ -- Predicting the Semantic Changes in Words across Corpora by Context Swapping
• Authors: Taichi Aida, Danushka Bollegala
• Abstract summary: We consider the problem of predicting whether a given target word, $w$, changes its meaning between two different text corpora. We propose an unsupervised method that randomly swaps contexts between $mathcalC$ and $mathcalC$. Our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task.
• Score: 36.10628959436778
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Meanings of words change over time and across domains. Detecting the semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. We consider the problem of predicting whether a given target word, $w$, changes its meaning between two different text corpora, $\mathcal{C}_1$ and $\mathcal{C}_2$. For this purpose, we propose $\textit{Swapping-based Semantic Change Detection}$ (SSCD), an unsupervised method that randomly swaps contexts between $\mathcal{C}_1$ and $\mathcal{C}_2$ where $w$ occurs. We then look at the distribution of contextualised word embeddings of $w$, obtained from a pretrained masked language model (MLM), representing the meaning of $w$ in its occurrence contexts in $\mathcal{C}_1$ and $\mathcal{C}_2$. Intuitively, if the meaning of $w$ does not change between $\mathcal{C}_1$ and $\mathcal{C}_2$, we would expect the distributions of contextualised word embeddings of $w$ to remain the same before and after this random swapping process. Despite its simplicity, we demonstrate that even by using pretrained MLMs without any fine-tuning, our proposed context swapping method accurately predicts the semantic changes of words in four languages (English, German, Swedish, and Latin) and across different time spans (over 50 years and about five years). Moreover, our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task. Source code is available at https://github.com/a1da4/svp-swap .

Related papers

• Creating an AI Observer: Generative Semantic Workspaces [4.031100721019478]
  We introduce the $textbf[G]$enerative $textbf[S]$emantic $textbf[W]$orkspace (GSW)) GSW creates a generative-style Semantic framework, as opposed to a traditionally predefined set of lexicon labels.
  arXiv  Detail & Related papers  (2024-06-07T00:09:13Z)
• Transformer In-Context Learning for Categorical Data [51.23121284812406]
  We extend research on understanding Transformers through the lens of in-context learning with functional data by considering categorical outcomes, nonlinear underlying models, and nonlinear attention. We present what is believed to be the first real-world demonstration of this few-shot-learning methodology, using the ImageNet dataset.
  arXiv  Detail & Related papers  (2024-05-27T15:03:21Z)
• Semantic Text Transmission via Prediction with Small Language Models: Cost-Similarity Trade-off [7.666188363531336]
  We exploit language's inherent correlations and predictability to constrain transmission costs by allowing the destination to predict or complete words. We obtain $(barc, bars)$ pairs for neural language and first-order Markov chain-based small language models. We demonstrate that, when communication occurs over a noiseless channel, the threshold policy achieves a higher $bars$ for a given $barc$ than the periodic policy.
  arXiv  Detail & Related papers  (2024-03-01T05:20:16Z)
• A Semantic Distance Metric Learning approach for Lexical Semantic Change Detection [30.563130208194977]
  A Lexical Semantic Change Detection (SCD) task involves predicting whether a given target word, $w$, changes its meaning between two different text corpora. We propose a supervised two-staged SCD method that uses existing Word-in-Context (WiC) datasets. Experimental results on multiple benchmark datasets for SCD show that our proposed method achieves strong performance in multiple languages.
  arXiv  Detail & Related papers  (2024-03-01T02:09:25Z)
• HappyMap: A Generalized Multi-calibration Method [23.086009024383024]
  Multi-calibration is a powerful and evolving concept originating in the field of algorithmic fairness. In this work, we view the term $(f(x)-y)$ as just one specific mapping, and explore the power of an enriched class of mappings. We propose textitHappyMap, a generalization of multi-calibration, which yields a wide range of new applications.
  arXiv  Detail & Related papers  (2023-03-08T05:05:01Z)
• Phase Transitions in the Detection of Correlated Databases [12.010807505655238]
  We study the problem of detecting the correlation between two Gaussian databases $mathsfXinmathbbRntimes d$ and $mathsfYntimes d$, each composed of $n$ users with $d$ features. This problem is relevant in the analysis of social media, computational biology, etc.
  arXiv  Detail & Related papers  (2023-02-07T10:39:44Z)
• Reward-Mixing MDPs with a Few Latent Contexts are Learnable [75.17357040707347]
  We consider episodic reinforcement learning in reward-mixing Markov decision processes (RMMDPs) Our goal is to learn a near-optimal policy that nearly maximizes the $H$ time-step cumulative rewards in such a model.
  arXiv  Detail & Related papers  (2022-10-05T22:52:00Z)
• Supervised Training of Conditional Monge Maps [107.78770597815242]
  Optimal transport (OT) theory describes general principles to define and select, among many possible choices, the most efficient way to map a probability measure onto another. We introduce CondOT, a multi-task approach to estimate a family of OT maps conditioned on a context variable. We demonstrate the ability of CondOT to infer the effect of an arbitrary combination of genetic or therapeutic perturbations on single cells.
  arXiv  Detail & Related papers  (2022-06-28T19:34:44Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• Taking a hint: How to leverage loss predictors in contextual bandits? [63.546913998407405]
  We study learning in contextual bandits with the help of loss predictors. We show that the optimal regret is $mathcalO(minsqrtT, sqrtmathcalETfrac13)$ when $mathcalE$ is known.
  arXiv  Detail & Related papers  (2020-03-04T07:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.