LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum

• URL: http://arxiv.org/abs/2601.01684v1
• Date: Sun, 04 Jan 2026 22:42:20 GMT
• Title: LACONIC: Dense-Level Effectiveness for Scalable Sparse Retrieval via a Two-Phase Training Curriculum
• Authors: Zhichao Xu, Shengyao Zhuang, Crystina Zhang, Xueguang Ma, Yijun Tian, Maitrey Mehta, Jimmy Lin, Vivek Srikumar,
• Abstract summary: LACONIC is a family of learned sparse retrievers based on the Llama-3 architecture.<n>The 8B variant achieves a state-of-the-art 60.2 nDCG on the MTEB Retrieval benchmark, ranking 15th on the leaderboard.
• Score: 73.82125917416067
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While dense retrieval models have become the standard for state-of-the-art information retrieval, their deployment is often constrained by high memory requirements and reliance on GPU accelerators for vector similarity search. Learned sparse retrieval offers a compelling alternative by enabling efficient search via inverted indices, yet it has historically received less attention than dense approaches. In this report, we introduce LACONIC, a family of learned sparse retrievers based on the Llama-3 architecture (1B, 3B, and 8B). We propose a streamlined two-phase training curriculum consisting of (1) weakly supervised pre-finetuning to adapt causal LLMs for bidirectional contextualization and (2) high-signal finetuning using curated hard negatives. Our results demonstrate that LACONIC effectively bridges the performance gap with dense models: the 8B variant achieves a state-of-the-art 60.2 nDCG on the MTEB Retrieval benchmark, ranking 15th on the leaderboard as of January 1, 2026, while utilizing 71\% less index memory than an equivalent dense model. By delivering high retrieval effectiveness on commodity CPU hardware with a fraction of the compute budget required by competing models, LACONIC provides a scalable and efficient solution for real-world search applications.

Related papers

• Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models [28.80331720382804]
  We introduce Orion, a training framework that enables compact models to perform iterative retrieval through learned search strategies.<n>Orion combines synthetic trajectory generation and supervised fine-tuning to encourage diverse exploration patterns in models.<n>Despite using only 3% of the training data available, our 1.2B model achieves 77.6% success on SciFact.
  arXiv  Detail & Related papers  (2025-11-10T19:49:55Z)
• Blending Learning to Rank and Dense Representations for Efficient and Effective Cascades [18.081997057477178]
  We investigate the exploitation of both lexical and neural relevance signals for ad-hoc passage retrieval.<n>We employ a pipelined architecture where a dense neural retriever serves as the first stage and performs a nearest-neighbor search.<n>We achieve a boost in nDCG@10 of up to 11% with an increase in average query latency of only 4.3%.
  arXiv  Detail & Related papers  (2025-10-18T08:16:48Z)
• AHDMIL: Asymmetric Hierarchical Distillation Multi-Instance Learning for Fast and Accurate Whole-Slide Image Classification [51.525891360380285]
  AHDMIL is an Asymmetric Hierarchical Distillation Multi-Instance Learning framework.<n>It eliminates irrelevant patches through a two-step training process.<n>It consistently outperforms previous state-of-the-art methods in both classification performance and inference speed.
  arXiv  Detail & Related papers  (2025-08-07T07:47:16Z)
• CSPLADE: Learned Sparse Retrieval with Causal Language Models [13.999080540889494]
  We identify two challenges in training large language models (LLM) for Learned sparse retrieval (LSR)<n>We propose two corresponding techniques: (1) a lightweight adaptation training phase to eliminate training instability; (2) two model variants to enable bidirectional information.<n>With these techniques, we are able to train LSR models with 8B scale LLM, and achieve competitive retrieval performance with reduced index size.
  arXiv  Detail & Related papers  (2025-04-15T02:31:34Z)
• Efficient Architecture Search via Bi-level Data Pruning [70.29970746807882]
  This work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization. We introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric. Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50%.
  arXiv  Detail & Related papers  (2023-12-21T02:48:44Z)
• DiffNAS: Bootstrapping Diffusion Models by Prompting for Better Architectures [63.12993314908957]
  We propose a base model search approach, denoted "DiffNAS" We leverage GPT-4 as a supernet to expedite the search, supplemented with a search memory to enhance the results. Rigorous experimentation corroborates that our algorithm can augment the search efficiency by 2 times under GPT-based scenarios.
  arXiv  Detail & Related papers  (2023-10-07T09:10:28Z)
• Improving Dual-Encoder Training through Dynamic Indexes for Negative Mining [61.09807522366773]
  We introduce an algorithm that approximates the softmax with provable bounds and that dynamically maintains the tree. In our study on datasets with over twenty million targets, our approach cuts error by half in relation to oracle brute-force negative mining.
  arXiv  Detail & Related papers  (2023-03-27T15:18:32Z)
• An Efficiency Study for SPLADE Models [5.725475501578801]
  In this paper, we focus on improving the efficiency of the SPLADE model. We propose several techniques including L1 regularization for queries, a separation of document/ encoders, a FLOPS-regularized middle-training, and the use of faster query encoders.
  arXiv  Detail & Related papers  (2022-07-08T11:42:05Z)
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval [11.38022203865326]
  SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. We modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. Overall, SPLADE is considerably improved with more than $9$% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.
  arXiv  Detail & Related papers  (2021-09-21T10:43:42Z)
• MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation [153.56211546576978]
  In this work, we propose that better soft targets with higher compatibil-ity can be generated by using a label generator. We can employ the meta-learning technique to optimize this label generator. The experiments are conducted on two standard classificationbenchmarks, namely CIFAR-100 and ILSVRC2012.
  arXiv  Detail & Related papers  (2020-08-27T13:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.