Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings

• URL: http://arxiv.org/abs/2204.12386v1
• Date: Tue, 26 Apr 2022 15:41:06 GMT
• Title: Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings
• Authors: Danushka Bollegala
• Abstract summary: We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding. We propose two emphunsupervised methods to learn the optimal concatenation weights for creating meta-embeddings.
• Score: 15.900069711477542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. Prior work on meta-embedding has repeatedly discovered that simple vector concatenation of the source embeddings to be a competitive baseline. However, it remains unclear as to why and when simple vector concatenation can produce accurate meta-embeddings. We show that weighted concatenation can be seen as a spectrum matching operation between each source embedding and the meta-embedding, minimising the pairwise inner-product loss. Following this theoretical analysis, we propose two \emph{unsupervised} methods to learn the optimal concatenation weights for creating meta-embeddings from a given set of source embeddings. Experimental results on multiple benchmark datasets show that the proposed weighted concatenated meta-embedding methods outperform previously proposed meta-embedding learning methods.

Related papers

• Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment [69.67015515485349]
  We propose AutoRegEmbed, a contrastive learning method built on embedding conditional probability distributions. We show that our method significantly outperforms traditional contrastive learning approaches.
  arXiv  Detail & Related papers  (2025-02-17T03:36:25Z)
• Context-Aware Hierarchical Merging for Long Document Summarization [56.96619074316232]
  We propose different approaches to enrich hierarchical merging with context from the source document. Experimental results on datasets representing legal and narrative domains show that contextual augmentation consistently outperforms zero-shot and hierarchical merging baselines.
  arXiv  Detail & Related papers  (2025-02-03T01:14:31Z)
• Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses [14.088007380798635]
  We introduce a novel approach incorporating two-phase training to discover contextual embeddings of input sequences. This technique not only facilitates the design of a scalable model but also preserves interpretability. Our experimental findings revealed that the proposed method yields competitive performance compared to the previous approaches.
  arXiv  Detail & Related papers  (2025-01-31T10:39:04Z)
• Coarse-to-Fine Lightweight Meta-Embedding for ID-Based Recommendation [13.732081010190962]
  We develop a novel graph neural networks (GNNs) based recommender where each user and item serves as the node. In contrast to coarse-grained semantics, fine-grained semantics are well captured through sparse meta-embeddings. We propose a weight bridging update strategy that focuses on matching each coarse-grained meta-embedding with several fine-grained meta-ems based on the users/items' semantics.
  arXiv  Detail & Related papers  (2025-01-21T03:56:23Z)
• Cool-Fusion: Fuse Large Language Models without Training [73.17551121242602]
  emphCool-Fusion is a method that does not require any type of training like the ensemble approaches. emphCool-Fusion increases accuracy from three strong source LLMs by a significant 8%-17.8%.
  arXiv  Detail & Related papers  (2024-07-29T09:02:19Z)
• LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution [76.615287796753]
  We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models. This is achieved by combining contextual information with knowledge from structured lexical resources. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-11T21:25:56Z)
• Out-of-Manifold Regularization in Contextual Embedding Space for Text Classification [22.931314501371805]
  We propose a new approach to finding and regularizing the remainder of the space, referred to as out-of-manifold. We synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold.
  arXiv  Detail & Related papers  (2021-05-14T10:17:59Z)
• SemGloVe: Semantic Co-occurrences for GloVe from BERT [55.420035541274444]
  GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. We propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings.
  arXiv  Detail & Related papers  (2020-12-30T15:38:26Z)
• Learning Efficient Task-Specific Meta-Embeddings with Word Prisms [17.288765083303243]
  We introduce word prisms: a simple and efficient meta-embedding method that learns to combine source embeddings according to the task at hand. We evaluate word prisms in comparison to other meta-embedding methods on six extrinsic evaluations and observe that word prisms offer improvements on all tasks.
  arXiv  Detail & Related papers  (2020-11-05T16:08:50Z)
• Inferential Text Generation with Multiple Knowledge Sources and Meta-Learning [117.23425857240679]
  We study the problem of generating inferential texts of events for a variety of commonsense like textitif-else relations. Existing approaches typically use limited evidence from training examples and learn for each relation individually. In this work, we use multiple knowledge sources as fuels for the model.
  arXiv  Detail & Related papers  (2020-04-07T01:49:18Z)
• A Common Semantic Space for Monolingual and Cross-Lingual Meta-Embeddings [10.871587311621974]
  This paper presents a new technique for creating monolingual and cross-lingual meta-embeddings. Existing word vectors are projected to a common semantic space using linear transformations and averaging. The resulting cross-lingual meta-embeddings also exhibit excellent cross-lingual transfer learning capabilities.
  arXiv  Detail & Related papers  (2020-01-17T15:42:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.