Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection

• URL: http://arxiv.org/abs/2106.07947v1
• Date: Tue, 15 Jun 2021 08:02:42 GMT
• Title: Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection
• Authors: Yixiao Wang, Zied Bouraoui, Luis Espinosa Anke, Steven Schockaert
• Abstract summary: We propose a method for learning word representations that follows this basic strategy. We take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts. We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.
• Score: 46.97185212695267
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the long-standing challenges in lexical semantics consists in learning representations of words which reflect their semantic properties. The remarkable success of word embeddings for this purpose suggests that high-quality representations can be obtained by summarizing the sentence contexts of word mentions. In this paper, we propose a method for learning word representations that follows this basic strategy, but differs from standard word embeddings in two important ways. First, we take advantage of contextualized language models (CLMs) rather than bags of word vectors to encode contexts. Second, rather than learning a word vector directly, we use a topic model to partition the contexts in which words appear, and then learn different topic-specific vectors for each word. Finally, we use a task-specific supervision signal to make a soft selection of the resulting vectors. We show that this simple strategy leads to high-quality word vectors, which are more predictive of semantic properties than word embeddings and existing CLM-based strategies.

Related papers

• Visual Prompt Selection for In-Context Learning Segmentation [77.15684360470152]
  In this paper, we focus on rethinking and improving the example selection strategy. We first demonstrate that ICL-based segmentation models are sensitive to different contexts. Furthermore, empirical evidence indicates that the diversity of contextual prompts plays a crucial role in guiding segmentation.
  arXiv  Detail & Related papers  (2024-07-14T15:02:54Z)
• Vocabulary-free Image Classification and Semantic Segmentation [71.78089106671581]
  We introduce the Vocabulary-free Image Classification (VIC) task, which aims to assign a class from an un-constrained language-induced semantic space to an input image without needing a known vocabulary. VIC is challenging due to the vastness of the semantic space, which contains millions of concepts, including fine-grained categories. We propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database.
  arXiv  Detail & Related papers  (2024-04-16T19:27:21Z)
• Searching for Discriminative Words in Multidimensional Continuous Feature Space [0.0]
  We propose a novel method to extract discriminative keywords from documents. We show how different discriminative metrics influence the overall results. We conclude that word feature vectors can substantially improve the topical inference of documents' meaning.
  arXiv  Detail & Related papers  (2022-11-26T18:05:11Z)
• Word2Box: Learning Word Representation Using Box Embeddings [28.080105878687185]
  Learning vector representations for words is one of the most fundamental topics in NLP. Our model, Word2Box, takes a region-based approach to the problem of word representation, representing words as $n$-dimensional rectangles. We demonstrate improved performance on various word similarity tasks, particularly on less common words.
  arXiv  Detail & Related papers  (2021-06-28T01:17:11Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Modelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings [46.49372320363155]
  We show how the contextualised embeddings predicted by BERT can be used to produce high-quality word vectors. We find that a simple strategy of averaging the contextualised embeddings of masked word mentions leads to vectors that outperform the static word vectors.
  arXiv  Detail & Related papers  (2020-12-04T14:03:03Z)
• Robust and Consistent Estimation of Word Embedding for Bangla Language by fine-tuning Word2Vec Model [1.2691047660244335]
  We analyze word2vec model for learning word vectors and present the most effective word embedding for Bangla language. We cluster the word vectors to examine the relational similarity of words for intrinsic evaluation and also use different word embeddings as the feature of news article for extrinsic evaluation.
  arXiv  Detail & Related papers  (2020-10-26T08:00:48Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Using Holographically Compressed Embeddings in Question Answering [0.0]
  This research employs holographic compression of pre-trained embeddings to represent a token, its part-of-speech, and named entity type. The implementation, in a modified question answering recurrent deep learning network, shows that semantic relationships are preserved, and yields strong performance.
  arXiv  Detail & Related papers  (2020-07-14T18:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.