Unsupervised Lexical Substitution with Decontextualised Embeddings
- URL: http://arxiv.org/abs/2209.08236v1
- Date: Sat, 17 Sep 2022 03:51:47 GMT
- Title: Unsupervised Lexical Substitution with Decontextualised Embeddings
- Authors: Takashi Wada, Timothy Baldwin, Yuji Matsumoto, Jey Han Lau
- Abstract summary: We propose a new unsupervised method for lexical substitution using pre-trained language models.
Our method retrieves substitutes based on the similarity of contextualised and decontextualised word embeddings.
We conduct experiments in English and Italian, and show that our method substantially outperforms strong baselines.
- Score: 48.00929769805882
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We propose a new unsupervised method for lexical substitution using
pre-trained language models. Compared to previous approaches that use the
generative capability of language models to predict substitutes, our method
retrieves substitutes based on the similarity of contextualised and
decontextualised word embeddings, i.e. the average contextual representation of
a word in multiple contexts. We conduct experiments in English and Italian, and
show that our method substantially outperforms strong baselines and establishes
a new state-of-the-art without any explicit supervision or fine-tuning. We
further show that our method performs particularly well at predicting
low-frequency substitutes, and also generates a diverse list of substitute
candidates, reducing morphophonetic or morphosyntactic biases induced by
article-noun agreement.
Related papers
- Unsupervised Lexical Simplification with Context Augmentation [55.318201742039]
Given a target word and its context, our method generates substitutes based on the target context and additional contexts sampled from monolingual data.
We conduct experiments in English, Portuguese, and Spanish on the TSAR-2022 shared task, and show that our model substantially outperforms other unsupervised systems across all languages.
arXiv Detail & Related papers (2023-11-01T05:48:05Z) - ParaLS: Lexical Substitution via Pretrained Paraphraser [18.929859707202517]
This study explores how to generate the substitute candidates from a paraphraser.
We propose two simple decoding strategies that focus on the variations of the target word during decoding.
arXiv Detail & Related papers (2023-05-14T12:49:16Z) - Contextualized language models for semantic change detection: lessons
learned [4.436724861363513]
We present a qualitative analysis of the outputs of contextualized embedding-based methods for detecting diachronic semantic change.
Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift.
Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance.
arXiv Detail & Related papers (2022-08-31T23:35:24Z) - LexSubCon: Integrating Knowledge from Lexical Resources into Contextual
Embeddings for Lexical Substitution [76.615287796753]
We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models.
This is achieved by combining contextual information with knowledge from structured lexical resources.
Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets.
arXiv Detail & Related papers (2021-07-11T21:25:56Z) - Obtaining Better Static Word Embeddings Using Contextual Embedding
Models [53.86080627007695]
Our proposed distillation method is a simple extension of CBOW-based training.
As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings.
arXiv Detail & Related papers (2021-06-08T12:59:32Z) - Denoising Word Embeddings by Averaging in a Shared Space [34.175826109538676]
We introduce a new approach for smoothing and improving the quality of word embeddings.
We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure.
As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.
arXiv Detail & Related papers (2021-06-05T19:49:02Z) - Unsupervised Word Translation Pairing using Refinement based Point Set
Registration [8.568050813210823]
Cross-lingual alignment of word embeddings play an important role in knowledge transfer across languages.
Current unsupervised approaches rely on similarities in geometric structure of word embedding spaces across languages.
This paper proposes BioSpere, a novel framework for unsupervised mapping of bi-lingual word embeddings onto a shared vector space.
arXiv Detail & Related papers (2020-11-26T09:51:29Z) - A Comparative Study of Lexical Substitution Approaches based on Neural
Language Models [117.96628873753123]
We present a large-scale comparative study of popular neural language and masked language models.
We show that already competitive results achieved by SOTA LMs/MLMs can be further improved if information about the target word is injected properly.
arXiv Detail & Related papers (2020-05-29T18:43:22Z) - Analysing Lexical Semantic Change with Contextualised Word
Representations [7.071298726856781]
We propose a novel method that exploits the BERT neural language model to obtain representations of word usages.
We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements.
arXiv Detail & Related papers (2020-04-29T12:18:14Z) - A Probabilistic Formulation of Unsupervised Text Style Transfer [128.80213211598752]
We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques.
By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion.
arXiv Detail & Related papers (2020-02-10T16:20:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.