Blind signal decomposition of various word embeddings based on join and individual variance explained

• URL: http://arxiv.org/abs/2011.14496v1
• Date: Mon, 30 Nov 2020 01:36:29 GMT
• Title: Blind signal decomposition of various word embeddings based on join and individual variance explained
• Authors: Yikai Wang and Weijian Li
• Abstract summary: We propose to use a novel joint signal separation method - JIVE to jointly decompose various trained word embeddings into joint and individual components. We conducted empirical study on word2vec, FastText and GLoVE trained on different corpus and with different dimensions. We found that by mapping different word embeddings into the joint component, sentiment performance can be greatly improved for the original word embeddings with lower performance.
• Score: 11.542392473831672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, natural language processing (NLP) has become one of the most important areas with various applications in human's life. As the most fundamental task, the field of word embedding still requires more attention and research. Currently, existing works about word embedding are focusing on proposing novel embedding algorithms and dimension reduction techniques on well-trained word embeddings. In this paper, we propose to use a novel joint signal separation method - JIVE to jointly decompose various trained word embeddings into joint and individual components. Through this decomposition framework, we can easily investigate the similarity and difference among different word embeddings. We conducted extensive empirical study on word2vec, FastText and GLoVE trained on different corpus and with different dimensions. We compared the performance of different decomposed components based on sentiment analysis on Twitter and Stanford sentiment treebank. We found that by mapping different word embeddings into the joint component, sentiment performance can be greatly improved for the original word embeddings with lower performance. Moreover, we found that by concatenating different components together, the same model can achieve better performance. These findings provide great insights into the word embeddings and our work offer a new of generating word embeddings by fusing.

Related papers

• Relational Sentence Embedding for Flexible Semantic Matching [86.21393054423355]
  We present Sentence Embedding (RSE), a new paradigm to discover further the potential of sentence embeddings. RSE is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art embedding methods.
  arXiv  Detail & Related papers  (2022-12-17T05:25:17Z)
• Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation [70.58243648754507]
  We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR) Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic similarity and transfer tasks.
  arXiv  Detail & Related papers  (2022-10-18T11:37:36Z)
• Human-in-the-Loop Refinement of Word Embeddings [0.0]
  We propose a system that incorporates an adaptation of word embedding post-processing, which we call "interactive refitting" Our approach allows a human to identify and address potential quality issues with word embeddings interactively. It also allows for better insight into what effect word embeddings, and refinements to word embeddings, have on machine learning pipelines.
  arXiv  Detail & Related papers  (2021-10-06T16:10:32Z)
• A Survey On Neural Word Embeddings [0.4822598110892847]
  The study of meaning in natural language processing relies on the distributional hypothesis. The revolutionary idea of distributed representation for a concept is close to the working of a human mind. Neural word embeddings transformed the whole field of NLP by introducing substantial improvements in all NLP tasks.
  arXiv  Detail & Related papers  (2021-10-05T03:37:57Z)
• LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution [76.615287796753]
  We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models. This is achieved by combining contextual information with knowledge from structured lexical resources. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-11T21:25:56Z)
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora [63.5286019659504]
  We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus. Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence.
  arXiv  Detail & Related papers  (2020-10-27T22:24:01Z)
• Interactive Re-Fitting as a Technique for Improving Word Embeddings [0.0]
  We make it possible for humans to adjust portions of a word embedding space by moving sets of words closer to one another. Our approach allows users to trigger selective post-processing as they interact with and assess potential bias in word embeddings.
  arXiv  Detail & Related papers  (2020-09-30T21:54:22Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Comparative Analysis of Word Embeddings for Capturing Word Similarities [0.0]
  Distributed language representation has become the most widely used technique for language representation in various natural language processing tasks. Most of the natural language processing models that are based on deep learning techniques use already pre-trained distributed word representations, commonly called word embeddings. selecting the appropriate word embeddings is a perplexing task since the projected embedding space is not intuitive to humans.
  arXiv  Detail & Related papers  (2020-05-08T01:16:03Z)
• Compass-aligned Distributional Embeddings for Studying Semantic Differences across Corpora [14.993021283916008]
  We present a framework to support cross-corpora language studies with word embeddings. CADE is the core component of our framework and solves the key problem of aligning the embeddings generated from different corpora. The results of our experiments suggest that CADE achieves state-of-the-art or superior performance on tasks where several competing approaches are available.
  arXiv  Detail & Related papers  (2020-04-13T15:46:47Z)
• Multiplex Word Embeddings for Selectional Preference Acquisition [70.33531759861111]
  We propose a multiplex word embedding model, which can be easily extended according to various relations among words. Our model can effectively distinguish words with respect to different relations without introducing unnecessary sparseness.
  arXiv  Detail & Related papers  (2020-01-09T04:47:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.