Revisiting Word Embeddings in the LLM Era

• URL: http://arxiv.org/abs/2502.19607v2
• Date: Sat, 01 Mar 2025 19:33:15 GMT
• Title: Revisiting Word Embeddings in the LLM Era
• Authors: Yash Mahajan, Matthew Freestone, Sathyanarayanan Aakur, Santu Karmaker,
• Abstract summary: Large Language Models (LLMs) have recently shown remarkable advancement in various NLP tasks.<n>We compare classical decontextualized and contextualized word embeddings with the same for LLM-induced embeddings.<n>Our results show that LLMs cluster semantically related words more tightly and perform better on analogy tasks in decontextualized settings.
• Score: 0.2999888908665658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have recently shown remarkable advancement in various NLP tasks. As such, a popular trend has emerged lately where NLP researchers extract word/sentence/document embeddings from these large decoder-only models and use them for various inference tasks with promising results. However, it is still unclear whether the performance improvement of LLM-induced embeddings is merely because of scale or whether underlying embeddings they produce significantly differ from classical encoding models like Word2Vec, GloVe, Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This is the central question we investigate in the paper by systematically comparing classical decontextualized and contextualized word embeddings with the same for LLM-induced embeddings. Our results show that LLMs cluster semantically related words more tightly and perform better on analogy tasks in decontextualized settings. However, in contextualized settings, classical models like SimCSE often outperform LLMs in sentence-level similarity assessment tasks, highlighting their continued relevance for fine-grained semantics.

Related papers

• Evaluating how LLM annotations represent diverse views on contentious topics [3.405231040967506]
  We show how generative large language models (LLMs) represent diverse viewpoints on contentious labeling tasks. Our findings suggest that when using LLMs to annotate data, under-representing the views of particular groups is not a substantial concern.
  arXiv  Detail & Related papers  (2025-03-29T22:53:15Z)
• Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models [60.00178316095646]
  Sentence embedding is essential for many NLP tasks, with contrastive learning methods achieving strong performance using datasets like NLI. Recent studies leverage large language models (LLMs) to generate sentence pairs, reducing annotation dependency. We propose a method for controlling the generation direction of LLMs in the latent space. Unlike unconstrained generation, the controlled approach ensures meaningful semantic divergence. Experiments on multiple benchmarks demonstrate that our method achieves new SOTA performance with a modest cost in ranking sentence synthesis.
  arXiv  Detail & Related papers  (2025-02-19T12:07:53Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs)<n>We find that fine-tuning existing text embedding models on LLM-generated texts yields excellent classification accuracy.<n>We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• Rethinking Semantic Parsing for Large Language Models: Enhancing LLM Performance with Semantic Hints [20.844061807562436]
  We propose SENSE, a novel prompting approach that embeds semantic hints within the prompt. Experiments show that SENSE consistently improves LLMs' performance across various tasks.
  arXiv  Detail & Related papers  (2024-09-22T14:35:09Z)
• LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification [34.9210323553677]
  We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications.
  arXiv  Detail & Related papers  (2024-08-06T15:55:05Z)
• Analyzing the Role of Semantic Representations in the Era of Large Language Models [104.18157036880287]
  We investigate the role of semantic representations in the era of large language models (LLMs) We propose an AMR-driven chain-of-thought prompting method, which we call AMRCoT. We find that it is difficult to predict which input examples AMR may help or hurt on, but errors tend to arise with multi-word expressions.
  arXiv  Detail & Related papers  (2024-05-02T17:32:59Z)
• Word Embeddings Revisited: Do LLMs Offer Something New? [2.822851601000061]
  Learning meaningful word embeddings is key to training a robust language model. The recent rise of Large Language Models (LLMs) has provided us with many new word/sentence/document embedding models.
  arXiv  Detail & Related papers  (2024-02-16T21:47:30Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings [4.545354973721937]
  We propose a novel model: backward dependency enhanced large language model (BeLLM) It learns sentence embeddings via transforming specific attention layers from uni- to bi-directional. It shows that auto-regressive LLMs benefit from backward dependencies for sentence embeddings.
  arXiv  Detail & Related papers  (2023-11-09T11:53:52Z)
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation [94.59630161324013]
  We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
  arXiv  Detail & Related papers  (2023-10-24T08:56:49Z)
• Waffling around for Performance: Visual Classification with Random Words and Broad Concepts [121.60918966567657]
  WaffleCLIP is a framework for zero-shot visual classification which simply replaces LLM-generated descriptors with random character and word descriptors. We conduct an extensive experimental study on the impact and shortcomings of additional semantics introduced with LLM-generated descriptors.
  arXiv  Detail & Related papers  (2023-06-12T17:59:48Z)
• Description-Based Text Similarity [59.552704474862004]
  We identify the need to search for texts based on abstract descriptions of their content. We propose an alternative model that significantly improves when used in standard nearest neighbor search.
  arXiv  Detail & Related papers  (2023-05-21T17:14:31Z)
• Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution [124.99894592871385]
  We present a large-scale comparative study of lexical substitution methods employing both old and most recent language models. We show that already competitive results achieved by SOTA LMs/MLMs can be further substantially improved if information about the target word is injected properly.
  arXiv  Detail & Related papers  (2022-06-07T16:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.