Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings

• URL: http://arxiv.org/abs/2204.12386v1
• Date: Tue, 26 Apr 2022 15:41:06 GMT
• Title: Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings
• Authors: Danushka Bollegala
• Abstract summary: We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding. We propose two emphunsupervised methods to learn the optimal concatenation weights for creating meta-embeddings.
• Score: 15.900069711477542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. Prior work on meta-embedding has repeatedly discovered that simple vector concatenation of the source embeddings to be a competitive baseline. However, it remains unclear as to why and when simple vector concatenation can produce accurate meta-embeddings. We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding, minimising the pairwise inner-product loss. Following this theoretical analysis, we propose two \emph{unsupervised} methods to learn the optimal concatenation weights for creating meta-embeddings from a given set of source embeddings. Experimental results on multiple benchmark datasets show that the proposed weighted concatenated meta-embedding methods outperform previously proposed meta-embedding learning methods.

Related papers

• Cool-Fusion: Fuse Large Language Models without Training [73.17551121242602]
  emphCool-Fusion is a method that does not require any type of training like the ensemble approaches. emphCool-Fusion increases accuracy from three strong source LLMs by a significant 8%-17.8%.
  arXiv  Detail & Related papers  (2024-07-29T09:02:19Z)
• Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
  We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation. We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
  arXiv  Detail & Related papers  (2024-02-28T16:35:52Z)
• Context-Aware Ensemble Learning for Time Series [11.716677452529114]
  We introduce a new approach using a meta learner that effectively combines the base model predictions via using a superset of the features that is the union of the base models' feature vectors instead of the predictions themselves. Our model does not use the predictions of the base models as inputs to a machine learning algorithm, but choose the best possible combination at each time step based on the state of the problem.
  arXiv  Detail & Related papers  (2022-11-30T10:36:13Z)
• LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution [76.615287796753]
  We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models. This is achieved by combining contextual information with knowledge from structured lexical resources. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-11T21:25:56Z)
• Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification [22.931314501371805]
  We propose a new approach to finding and regularizing the remainder of the space, referred to as out-of-manifold. We synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold.
  arXiv  Detail & Related papers  (2021-05-14T10:17:59Z)
• SemGloVe: Semantic Co-occurrences for GloVe from BERT [55.420035541274444]
  GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. We propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings.
  arXiv  Detail & Related papers  (2020-12-30T15:38:26Z)
• Learning Efficient Task-Specific Meta-Embeddings with Word Prisms [17.288765083303243]
  We introduce word prisms: a simple and efficient meta-embedding method that learns to combine source embeddings according to the task at hand. We evaluate word prisms in comparison to other meta-embedding methods on six extrinsic evaluations and observe that word prisms offer improvements on all tasks.
  arXiv  Detail & Related papers  (2020-11-05T16:08:50Z)
• A Comparative Study on Structural and Semantic Properties of Sentence Embeddings [77.34726150561087]
  We propose a set of experiments using a widely-used large-scale data set for relation extraction. We show that different embedding spaces have different degrees of strength for the structural and semantic properties. These results provide useful information for developing embedding-based relation extraction methods.
  arXiv  Detail & Related papers  (2020-09-23T15:45:32Z)
• Optimally Combining Classifiers for Semi-Supervised Learning [43.77365242185884]
  We propose a new semi-supervised learning method that is able to adaptively combine the strengths of Xgboost and transductive support vector machine. The experimental results on the UCI data sets and real commercial data set demonstrate the superior classification performance of our method over the five state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2020-06-07T09:28:34Z)
• Inferential Text Generation with Multiple Knowledge Sources and Meta-Learning [117.23425857240679]
  We study the problem of generating inferential texts of events for a variety of commonsense like textitif-else relations. Existing approaches typically use limited evidence from training examples and learn for each relation individually. In this work, we use multiple knowledge sources as fuels for the model.
  arXiv  Detail & Related papers  (2020-04-07T01:49:18Z)
• A Common Semantic Space for Monolingual and Cross-Lingual Meta-Embeddings [10.871587311621974]
  This paper presents a new technique for creating monolingual and cross-lingual meta-embeddings. Existing word vectors are projected to a common semantic space using linear transformations and averaging. The resulting cross-lingual meta-embeddings also exhibit excellent cross-lingual transfer learning capabilities.
  arXiv  Detail & Related papers  (2020-01-17T15:42:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.