The Embedder's Dilemma: LLMs Are Better, but at What Cost?

• URL: http://arxiv.org/abs/2608.12875v1
• Date: Thu, 13 Aug 2026 06:39:45 GMT
• Title: The Embedder's Dilemma: LLMs Are Better, but at What Cost?
• Abstract summary: We compare ten text-embedding models with 26 embedding models.<n>The two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro) and the best embedding model (77.2) differ by 0.4 points.<n>Our results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval.
• Score: 18.755750863102232
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Should you replace your text-embedding pipeline with a large language model? We answer this with a controlled, cost-aware comparison of ten LLMs across six families and 26 embedding models (118M to 14B parameters) on 37 tasks spanning classification, semantic textual similarity (STS), clustering, pair classification, and retrieval. In aggregate the two paradigms are effectively tied: the best LLM (Gemini 3.1 Pro, 77.6) and the best embedding model (77.2) differ by 0.4 points. Their strengths differ by task: LLMs lead on reasoning-heavy retrieval, embedding models lead on classification, and the two match on clustering, STS, and pair classification. Reaching that parity is expensive. An LLM costs up to 1,431x more than an embedding model of comparable quality (USD 154 vs. USD 0.11 per benchmark pass), and the open LLMs tested process tokens 2.5 to 736x more slowly on the same GPU. Reasoning tokens account for 28 to 81% of LLM inference cost; lower reasoning budgets preserve or improve retrieval quality for most models in our ablation. The Pareto frontier contains the leading embedding models and one LLM, Gemini 3.1 Pro. These results support a division of labour: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Our code, datasets, and results are publicly available at https://github.com/embeddings-benchmark/embedders-dilemma.

Related papers

• PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference [0.5568796330843819]
  PyroDash is a cost-aware framework for token-level SLM-LLM collaborative inference.<n>A Collaborate Engine sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff.
  arXiv  Detail & Related papers  (2026-07-22T16:14:26Z)
• Rethinking LLM Ensembling from the Perspective of Mixture Models [41.97945120072535]
  We propose the Mixture-like Ensemble (ME) for large language models (LLMs)<n>ME is mathematically equivalent to sampling from the ensemble distribution, but requires invoking only one model, making it 1.78x-2.68x faster than conventional ensemble.<n>Our findings open new avenues for efficient LLM ensembling and motivate further exploration of token-level routing strategies.
  arXiv  Detail & Related papers  (2026-05-01T05:31:18Z)
• A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction [0.0]
  This study evaluates the reliability of information extraction approaches from KRS documents using three strategies.<n>Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation.
  arXiv  Detail & Related papers  (2026-03-09T02:09:02Z)
• Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inference [7.865726406769634]
  Small Language Models (SLMs) offer dramatic cost savings yet lag substantially in accuracy.<n>We introduce LLM Shepherding, a framework that requests only a short prefix (a hint) from the LLM and provides it to SLM.<n>Shepherding generalizes both routing and cascading, and it achieves lower cost under oracle decision-making.
  arXiv  Detail & Related papers  (2026-01-29T18:52:54Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• Enhancing LLM Code Generation with Ensembles: A Similarity-Based Selection Approach [6.93983229112122]
  We propose an ensemble approach for large language models (LLMs) in code generation.<n>For voting, we compute syntactic and semantic similarity using CodeBLEU and behavioral equivalence.<n>We show through experiments that our ensemble approach consistently outperforms standalone LLMs.
  arXiv  Detail & Related papers  (2025-03-20T04:38:56Z)
• Preference Leakage: A Contamination Problem in LLM-as-a-judge [69.96778498636071]
  Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods.<n>In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators.
  arXiv  Detail & Related papers  (2025-02-03T17:13:03Z)
• Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval [55.63711219190506]
  Large language models (LLMs) often struggle with posing the right search queries. We introduce $underlineLe$arning to $underlineRe$trieve by $underlineT$rying (LeReT) LeReT can improve the absolute retrieval accuracy by up to 29% and the downstream generator evaluations by 17%.
  arXiv  Detail & Related papers  (2024-10-30T17:02:54Z)
• How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments [83.78240828340681]
  GAMA($gamma$)-Bench is a new framework for evaluating Large Language Models' Gaming Ability in Multi-Agent environments.<n>$gamma$-Bench includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to assess LLMs' performance.<n>Our results indicate GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought.
  arXiv  Detail & Related papers  (2024-03-18T14:04:47Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)
• You can't pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM [65.74934004876914]
  Retrieval-enhanced language models (LMs) condition their predictions on text retrieved from large external datastores. One such approach, the $k$NN-LM, interpolates any existing LM's predictions with the output of a $k$-nearest neighbors model. We empirically measure the effectiveness of our approach on two English language modeling datasets.
  arXiv  Detail & Related papers  (2022-10-28T02:57:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.