Denoising Word Embeddings by Averaging in a Shared Space

• URL: http://arxiv.org/abs/2106.02954v1
• Date: Sat, 5 Jun 2021 19:49:02 GMT
• Title: Denoising Word Embeddings by Averaging in a Shared Space
• Authors: Avi Caciularu, Ido Dagan, Jacob Goldberger
• Abstract summary: We introduce a new approach for smoothing and improving the quality of word embeddings. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.
• Score: 34.175826109538676
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.

Related papers

• Word Embedding with Neural Probabilistic Prior [24.893999575628452]
  We propose a probabilistic prior which can be seamlessly integrated with word embedding models. The structure of the proposed prior is simple and effective, and it can be easily implemented and flexibly plugged in.
  arXiv  Detail & Related papers  (2023-09-21T06:54:32Z)
• Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing [68.47787275021567]
  Cross-lingual semantic parsing transfers parsing capability from a high-resource language (e.g., English) to low-resource languages with scarce training data. We propose a new approach to cross-lingual semantic parsing by explicitly minimizing cross-lingual divergence between latent variables using Optimal Transport.
  arXiv  Detail & Related papers  (2023-07-09T04:52:31Z)
• Towards preserving word order importance through Forced Invalidation [80.33036864442182]
  We show that pre-trained language models are insensitive to word order. We propose Forced Invalidation to help preserve the importance of word order. Our experiments demonstrate that Forced Invalidation significantly improves the sensitivity of the models to word order.
  arXiv  Detail & Related papers  (2023-04-11T13:42:10Z)
• Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation [70.58243648754507]
  We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR) Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic similarity and transfer tasks.
  arXiv  Detail & Related papers  (2022-10-18T11:37:36Z)
• Robust Unsupervised Cross-Lingual Word Embedding using Domain Flow Interpolation [48.32604585839687]
  Previous adversarial approaches have shown promising results in inducing cross-lingual word embedding without parallel data. We propose to make use of a sequence of intermediate spaces for smooth bridging.
  arXiv  Detail & Related papers  (2022-10-07T04:37:47Z)
• Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost [5.672132510411465]
  State-of-the-art NLP systems represent inputs with word embeddings, but these are brittle when faced with Out-of-Vocabulary words. We follow the principle of mimick-like models to generate vectors for unseen words, by learning the behavior of pre-trained embeddings using only the surface form of words. We present a simple contrastive learning framework, LOVE, which extends the word representation of an existing pre-trained language model (such as BERT) and makes it robust to OOV with few additional parameters.
  arXiv  Detail & Related papers  (2022-03-15T13:11:07Z)
• Obtaining Better Static Word Embeddings Using Contextual Embedding Models [53.86080627007695]
  Our proposed distillation method is a simple extension of CBOW-based training. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings.
  arXiv  Detail & Related papers  (2021-06-08T12:59:32Z)
• Improving Chinese Segmentation-free Word Embedding With Unsupervised Association Measure [3.9435648520559177]
  segmentation-free word embedding model is proposed by collecting n-grams vocabulary via a novel unsupervised association measure called pointwise association with times information(PATI) The proposed method leverages more latent information from the corpus and thus is able to collect more valid n-grams that have stronger cohesion as embedding targets in unsegmented language data, such as Chinese texts.
  arXiv  Detail & Related papers  (2020-07-05T13:55:19Z)
• Robust Cross-lingual Embeddings from Parallel Sentences [65.85468628136927]
  We propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word representations. Our approach significantly improves crosslingual sentence retrieval performance over all other approaches. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task.
  arXiv  Detail & Related papers  (2019-12-28T16:18:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.