Does Generative Retrieval Overcome the Limitations of Dense Retrieval?

• URL: http://arxiv.org/abs/2509.22116v1
• Date: Fri, 26 Sep 2025 09:38:01 GMT
• Title: Does Generative Retrieval Overcome the Limitations of Dense Retrieval?
• Authors: Yingchen Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, Xueqi Cheng,
• Abstract summary: Generative retrieval (GR) has emerged as a new paradigm in neural information retrieval.<n>We investigate how GR fundamentally diverges from dense retrieval (DR) in both learning objectives and representational capacity.
• Score: 117.49076770892583
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative retrieval (GR) has emerged as a new paradigm in neural information retrieval, offering an alternative to dense retrieval (DR) by directly generating identifiers of relevant documents. In this paper, we theoretically and empirically investigate how GR fundamentally diverges from DR in both learning objectives and representational capacity. GR performs globally normalized maximum-likelihood optimization and encodes corpus and relevance information directly in the model parameters, whereas DR adopts locally normalized objectives and represents the corpus with external embeddings before computing similarity via a bilinear interaction. Our analysis suggests that, under scaling, GR can overcome the inherent limitations of DR, yielding two major benefits. First, with larger corpora, GR avoids the sharp performance degradation caused by the optimization drift induced by DR's local normalization. Second, with larger models, GR's representational capacity scales with parameter size, unconstrained by the global low-rank structure that limits DR. We validate these theoretical insights through controlled experiments on the Natural Questions and MS MARCO datasets, across varying negative sampling strategies, embedding dimensions, and model scales. But despite its theoretical advantages, GR does not universally outperform DR in practice. We outline directions to bridge the gap between GR's theoretical potential and practical performance, providing guidance for future research in scalable and robust generative retrieval.

Related papers

• Bringing Reasoning to Generative Recommendation Through the Lens of Cascaded Ranking [107.09842504618369]
  Generative Recommendation (GR) has become a promising end-to-end approach with high FLOPS utilization for resource-efficient recommendation.<n>We show that current GR models suffer from a critical textbfbias amplification issue, where token-level bias escalates as token generation progresses.<n>To combat the bias amplification issue, it is crucial for GR to 1) incorporate more heterogeneous information, and 2) allocate greater computational resources at each token generation step.
  arXiv  Detail & Related papers  (2026-02-03T16:10:54Z)
• GDR-learners: Orthogonal Learning of Generative Models for Potential Outcomes [50.228749840286895]
  We introduce a general suite of generative Neyman-orthogonal learners that estimate conditional distributions of potential outcomes.<n>Our proposed GDR-learners are flexible and can be instantiated with many state-of-the-art deep generative models.<n>Unlike the existing methods, our GDR-learners possess the properties of quasi-oracle efficiency and double robustness.
  arXiv  Detail & Related papers  (2025-09-26T21:35:28Z)
• On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization [52.76330545825083]
  Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs)<n>We identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training.<n>We develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens.
  arXiv  Detail & Related papers  (2025-05-24T18:58:51Z)
• DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management [18.953750405635393]
  Decoupled Group Reward Optimization (DGRO) is a general RL algorithm for Large Language Models (LLMs) reasoning.<n>We show that DGRO achieves state-of-the-art performance on the Logic dataset with an average accuracy of 96.9%, and demonstrates strong generalization across mathematical benchmarks.
  arXiv  Detail & Related papers  (2025-05-19T10:44:49Z)
• Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps [16.84310001807895]
  This paper introduces a model-agnostic approach that can be applied to A-RAG methods.<n>Specifically, we use cache access and parallel generation to speed up the prefilling and decoding stages respectively.
  arXiv  Detail & Related papers  (2025-05-19T05:39:38Z)
• Constrained Auto-Regressive Decoding Constrains Generative Retrieval [71.71161220261655]
  Generative retrieval seeks to replace traditional search index data structures with a single large-scale neural network.<n>In this paper, we examine the inherent limitations of constrained auto-regressive generation from two essential perspectives: constraints and beam search.
  arXiv  Detail & Related papers  (2025-04-14T06:54:49Z)
• An In-depth Investigation of Sparse Rate Reduction in Transformer-like Models [32.04194224236952]
  We propose an information-theoretic objective function called Sparse Rate Reduction (SRR) We show that SRR has a positive correlation coefficient and outperforms other baseline measures, such as path-norm and sharpness-based ones. We show that generalization can be improved using SRR as regularization on benchmark image classification datasets.
  arXiv  Detail & Related papers  (2024-11-26T07:44:57Z)
• How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval [80.54532535622988]
  We show that a generalizable dense retriever can be trained to achieve high accuracy in both supervised and zero-shot retrieval. DRAGON, our dense retriever trained with diverse augmentation, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations.
  arXiv  Detail & Related papers  (2023-02-15T03:53:26Z)
• Understanding Overparameterization in Generative Adversarial Networks [56.57403335510056]
  Generative Adversarial Networks (GANs) are used to train non- concave mini-max optimization problems. A theory has shown the importance of the gradient descent (GD) to globally optimal solutions. We show that in an overized GAN with a $1$-layer neural network generator and a linear discriminator, the GDA converges to a global saddle point of the underlying non- concave min-max problem.
  arXiv  Detail & Related papers  (2021-04-12T16:23:37Z)
• Learning Domain Invariant Representations for Generalizable Person Re-Identification [71.35292121563491]
  Generalizable person Re-Identification (ReID) has attracted growing attention in recent computer vision community. We introduce causality into person ReID and propose a novel generalizable framework, named Domain Invariant Representations for generalizable person Re-Identification (DIR-ReID)
  arXiv  Detail & Related papers  (2021-03-29T18:59:48Z)
• Restrained Generative Adversarial Network against Overfitting in Numeric Data Augmentation [9.265768052866786]
  Generative Adversarial Network (GAN) is one of the popular schemes to augment the image dataset. In our study we find the generator G in the GAN fails to generate numerical data in lower-dimensional spaces. We propose a theoretical restraint, independence on the loss function, to suppress the overfitting.
  arXiv  Detail & Related papers  (2020-10-26T13:01:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.