Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers

• URL: http://arxiv.org/abs/2609.13486v1
• Date: Fri, 11 Sep 2026 19:55:03 GMT
• Title: Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers
• Abstract summary: Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models.<n>We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR.<n>One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time.
• Score: 2.8881610726882827
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has shown that fine-tuning decoder-only large language models (LLMs) for retrieval yields strong first-stage retrievers, with effectiveness improving as backbones grow in size. However, every query and document must pass through the full model, so encoding cost increases with model size. Mixture-of-Experts (MoE) LLMs activate only a subset of parameters per token and are widely used to scale generative models, yet remain underexplored as retrievers. We systematically study MoE backbones for retrieval by training MoE and dense LLMs from several families using the same procedure, evaluating them across diverse datasets, and measuring query encoding time under the same serving configuration. We show that MoE retrievers outperform dense retrievers with comparable active parameter counts by up to 3.0 nDCG@10 points on BEIR. One of our strongest MoE retrievers matches an 8B dense retriever with 59% fewer active parameters and 18% lower query encoding time. We further show that the number of experts used for query encoding can be reduced without retraining or re-indexing, retaining more than 99% of retrieval effectiveness while reducing query encoding time by up to 26%. Recent rerankers provide only modest additional gains over strong MoE first stages, which often match or exceed the reranked configurations we evaluate. Together, these results show that MoE LLMs can be strong and efficient first-stage retrievers.

Related papers

• Select, Don't Train: The Benefits of Modular Entity Disambiguation with LLM-Based Selection [15.630318972190702]
  We present a systematic comparison of retrieval strategies for candidate generation under a shared LLM-based selection stage.<n>We show that a fully training-free BM25 retriever paired with an LLM selector reaches a new state of the art on the ZELDA benchmark.
  arXiv  Detail & Related papers  (2026-08-17T16:52:05Z)
• SAGE: Benchmarking and Improving Retrieval for Deep Research Agents [60.53966065867568]
  We introduce SAGE, a benchmark for scientific literature retrieval comprising 1,200 queries across four scientific domains, with a 200,000 paper retrieval corpus.<n>We evaluate six deep research agents and find that all systems struggle with reasoning-intensive retrieval.<n> BM25 significantly outperforms LLM-based retrievers by approximately 30%, as existing agents generate keyword-oriented sub-queries.
  arXiv  Detail & Related papers  (2026-02-05T18:25:24Z)
• Making Large Language Models Efficient Dense Retrievers [24.74245913953312]
  EffiR is a framework for developing efficient retrievers that performs large-scale compression through a coarse-to-fine strategy.<n>EffiR achieves substantial reductions in model size and inference cost while preserving the performance of full-size models.
  arXiv  Detail & Related papers  (2025-12-23T18:58:25Z)
• FreeRet: MLLMs as Training-Free Retrievers [21.04237443940747]
  FreeRet is a plug-and-play framework that turns any MLLM into a two-stage retriever.<n>On the MMEB and MMEB-V2 benchmarks, FreeRet substantially outperforms models trained on millions of pairs.
  arXiv  Detail & Related papers  (2025-09-29T11:28:42Z)
• Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion [0.0]
  Large Language Models (LLMs) have shown potential in generating hypothetical documents for query expansion.<n>We introduce a novel fusion ranking framework, Exp4Fuse, which enhances the performance of sparse retrievers.
  arXiv  Detail & Related papers  (2025-06-05T08:44:34Z)
• ReasonIR: Training Retrievers for Reasoning Tasks [139.54343970560103]
  ReasonIR-8B is the first retriever specifically trained for general reasoning tasks.<n>It achieves a new state-of-the-art of 29.9 nDCG@10 without reranker and 36.9 nDCG@10 with reranker on BRIGHT, a widely-used information retrieval benchmark.
  arXiv  Detail & Related papers  (2025-04-29T09:49:28Z)
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers [86.54316283425001]
  Large language models (LLMs) have demonstrated strong effectiveness and robustness while fine-tuned as dense retrievers.<n>LLMs offer better efficiency, but they often fail to generalize effectively with limited supervised fine-tuning data.<n>We introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers.
  arXiv  Detail & Related papers  (2025-02-25T18:59:07Z)
• Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling [42.67141329779589]
  Grouped Cross Attention can generalize to 1000 times the pre-training context length.<n>Experiments show that GCA-based models achieve near-perfect accuracy in passkey retrieval for 16M context lengths.
  arXiv  Detail & Related papers  (2024-10-02T15:18:34Z)
• Mistral-SPLADE: LLMs for better Learned Sparse Retrieval [7.652738829153342]
  We propose to use decoder-only model for learning semantic keyword expansion. We use Mistral as the backbone to develop our Learned Sparse Retriever similar to SPLADE. Our experiments support the hypothesis that a sparse retrieval model based on decoder only large language model (LLM) surpasses the performance of existing LSR systems.
  arXiv  Detail & Related papers  (2024-08-20T18:21:54Z)
• SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval [92.27387459751309]
  We provide SPRINT, a unified Python toolkit for evaluating neural sparse retrieval. We establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document.
  arXiv  Detail & Related papers  (2023-07-19T22:48:02Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)
• Query2doc: Query Expansion with Large Language Models [69.9707552694766]
  The proposed method first generates pseudo- documents by few-shot prompting large language models (LLMs) query2doc boosts the performance of BM25 by 3% to 15% on ad-hoc IR datasets. Our method also benefits state-of-the-art dense retrievers in terms of both in-domain and out-of-domain results.
  arXiv  Detail & Related papers  (2023-03-14T07:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.