Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings

• URL: http://arxiv.org/abs/2305.19092v1
• Date: Tue, 30 May 2023 14:53:44 GMT
• Title: Together We Make Sense -- Learning Meta-Sense Embeddings from Pretrained Static Sense Embeddings
• Authors: Haochen Luo, Yi Zhou, Danushka Bollegala
• Abstract summary: We propose the first-ever meta-sense embedding method -- Neighbour Preserving Meta-Sense Embeddings. Our proposed method can combine source sense embeddings that cover different sets of word senses. Experimental results on Word Sense Disambiguation (WSD) and Word-in-Context (WiC) tasks show that the proposed meta-sense embedding method consistently outperforms several competitive baselines.
• Score: 19.12036493733793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sense embedding learning methods learn multiple vectors for a given ambiguous word, corresponding to its different word senses. For this purpose, different methods have been proposed in prior work on sense embedding learning that use different sense inventories, sense-tagged corpora and learning methods. However, not all existing sense embeddings cover all senses of ambiguous words equally well due to the discrepancies in their training resources. To address this problem, we propose the first-ever meta-sense embedding method -- Neighbour Preserving Meta-Sense Embeddings, which learns meta-sense embeddings by combining multiple independently trained source sense embeddings such that the sense neighbourhoods computed from the source embeddings are preserved in the meta-embedding space. Our proposed method can combine source sense embeddings that cover different sets of word senses. Experimental results on Word Sense Disambiguation (WSD) and Word-in-Context (WiC) tasks show that the proposed meta-sense embedding method consistently outperforms several competitive baselines.

Related papers

• Semantic Specialization for Knowledge-based Word Sense Disambiguation [12.573927420408365]
  A promising approach for knowledge-based Word Sense Disambiguation (WSD) is to select the sense whose contextualized embeddings are closest to those computed for a target word in a given sentence. We propose a semantic specialization for WSD where contextualized embeddings are adapted to the WSD task using solely lexical knowledge.
  arXiv  Detail & Related papers  (2023-04-22T07:40:23Z)
• Word Sense Induction with Knowledge Distillation from BERT [6.88247391730482]
  This paper proposes a method to distill multiple word senses from a pre-trained language model (BERT) by using attention over the senses of a word in a context. Experiments on the contextual word similarity and sense induction tasks show that this method is superior to or competitive with state-of-the-art multi-sense embeddings.
  arXiv  Detail & Related papers  (2023-04-20T21:05:35Z)
• Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories [47.03271152494389]
  Word Sense Disambiguation aims to automatically identify the exact meaning of one word according to its context. Existing supervised models struggle to make correct predictions on rare word senses due to limited training data. We propose a gloss alignment algorithm that can align definition sentences with the same meaning from different sense inventories to collect rich lexical knowledge.
  arXiv  Detail & Related papers  (2021-10-27T00:04:33Z)
• Large Scale Substitution-based Word Sense Induction [48.49573297876054]
  We present a word-sense induction method based on pre-trained masked language models (MLMs), which can cheaply scale to large vocabularies and large corpora. The result is a corpus which is sense-tagged according to a corpus-derived sense inventory and where each sense is associated with indicative words. Evaluation on English Wikipedia that was sense-tagged using our method shows that both the induced senses, and the per-instance sense assignment, are of high quality even compared to WSD methods, such as Babelfy.
  arXiv  Detail & Related papers  (2021-10-14T19:40:37Z)
• Toward a Visual Concept Vocabulary for GAN Latent Space [74.12447538049537]
  This paper introduces a new method for building open-ended vocabularies of primitive visual concepts represented in a GAN's latent space. Our approach is built from three components: automatic identification of perceptually salient directions based on their layer selectivity; human annotation of these directions with free-form, compositional natural language descriptions. Experiments show that concepts learned with our approach are reliable and composable -- generalizing across classes, contexts, and observers.
  arXiv  Detail & Related papers  (2021-10-08T17:58:19Z)
• Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy [26.385418377513332]
  We propose Context Derived Embeddings of Senses (CDES) CDES extracts sense related information from contextualised embeddings and injects it into static embeddings to create sense-specific static embeddings. We show that CDES can accurately learn sense-specific static embeddings reporting comparable performance to the current state-of-the-art sense embeddings.
  arXiv  Detail & Related papers  (2021-10-05T17:50:48Z)
• Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge [62.46091695615262]
  We aim to extract commonsense knowledge to improve machine reading comprehension. We propose to represent relations implicitly by situating structured knowledge in a context. We employ a teacher-student paradigm to inject multiple types of contextualized knowledge into a student machine reader.
  arXiv  Detail & Related papers  (2020-09-12T17:20:01Z)
• Moving Down the Long Tail of Word Sense Disambiguation with Gloss-Informed Biencoders [79.38278330678965]
  A major obstacle in Word Sense Disambiguation (WSD) is that word senses are not uniformly distributed. We propose a bi-encoder model that independently embeds (1) the target word with its surrounding context and (2) the dictionary definition, or gloss, of each sense.
  arXiv  Detail & Related papers  (2020-05-06T04:21:45Z)
• Decomposing Word Embedding with the Capsule Network [23.294890047230584]
  We propose a capsule network-based method to Decompose the unsupervised word Embedding of an ambiguous word into context specific Sense embedding. With attention operations, CapsDecE2S integrates the word context to reconstruct the multiple morpheme-like vectors into the context-specific sense embedding. In this method, we convert the sense learning into a binary classification that explicitly learns the relation between senses by the label of matching and non-matching.
  arXiv  Detail & Related papers  (2020-04-07T06:37:27Z)
• Word Sense Disambiguation for 158 Languages using Word Embeddings Only [80.79437083582643]
  Disambiguation of word senses in context is easy for humans, but a major challenge for automatic approaches. We present a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory. We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings.
  arXiv  Detail & Related papers  (2020-03-14T14:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.