Predictive Prefetching for Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2605.17989v1
• Date: Mon, 18 May 2026 07:45:27 GMT
• Title: Predictive Prefetching for Retrieval-Augmented Generation
• Abstract summary: We propose an advanced asynchronous retrieval framework that enables prefetching aligned with evolving information needs.<n> Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token.
• Score: 12.088272012448328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) improves factual grounding in large language models but suffers from substantial latency due to synchronous retrieval. While recent work explores asynchronous retrieval, existing approaches rely on heuristic coordination between retrieval and generation and assume stable information demands during decoding that often break in complex, multi-domain settings. In this paper, we propose an advanced asynchronous retrieval framework that enables predictive prefetching aligned with evolving information needs. The framework explicitly predicts when retrieval should be triggered and what information should be retrieved using three components, a retrieval predictor, a context monitor, and a query generator, by exploiting semantic precursors in generation dynamics that emerge several tokens before uncertainty becomes critical. Experiments on multiple benchmarks demonstrate up to 43.5% end-to-end latency reduction and 62.4% improvement in time-to-first-token, while maintaining answer quality comparable to synchronous RAG baselines.

Related papers

• Bridging the Question-Answer Gap in Retrieval-Augmented Generation: Hypothetical Prompt Embeddings [42.348431760663544]
  Hypothetical Prompt Embeddings (HyPE) is a framework that shifts the generation of hypothetical content from query time to the indexing phase.<n>We show that HyPE can improve retrieval context precision by up to 42 percentage points and claim recall by up to 45 percentage points, compared to standard approaches.
  arXiv  Detail & Related papers  (2026-07-31T13:22:11Z)
• Integrating Chain-of-Thought into Generative Retrieval: A Preliminary Study [31.167867050243256]
  ThinkGR is a unified framework that interleaves chain-of-thought with docid generation, enabling iterative thinking and retrieval within a single generative process.<n> Experiments on four multi-hop retrieval benchmarks demonstrate that ThinkGR achieves state-of-the-art performance with an average improvement of +6.86%.
  arXiv  Detail & Related papers  (2026-05-21T11:48:35Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• Transforming External Knowledge into Triplets for Enhanced Retrieval in RAG of LLMs [55.78708003681562]
  Retrieval-Augmented Generation (RAG) mitigates hallucination in large language models (LLMs) by incorporating external knowledge during generation.<n>Existing RAG approaches typically retrieve and unstructured text fragments as context.<n>We propose Tri-RAG, a structured triplet-based retrieval framework that improves retrieval efficiency through reasoning-aligned context construction.
  arXiv  Detail & Related papers  (2026-04-14T11:36:29Z)
• RAG without Forgetting: Continual Query-Infused Key Memory [27.597076561986594]
  We propose Evolving Retrieval Memory (ERM), a training-free framework that transforms transient query-time gains into persistent retrieval improvements.<n>ERM updates the retrieval index through correctness-gated feedback, selectively attributes atomic expansion signals to the document keys they benefit, and progressively evolves keys via stable, norm-bounded updates.<n>We show that query and key expansion are theoretically equivalent under standard similarity functions and prove convergence of ERM's selective updates, amortizing optimal query expansion into a stable index with zero inference-time overhead.
  arXiv  Detail & Related papers  (2026-02-05T00:12:45Z)
• Re3: Learning to Balance Relevance & Recency for Temporal Information Retrieval [10.939002113975706]
  Temporal Information Retrieval is a critical yet unresolved task for modern search systems.<n>Re3 is a framework that balances semantic and temporal information through a query-aware gating mechanism.<n>On Re2Bench, Re3 achieves state-of-the-art results, leading in R@1 across all three subsets.
  arXiv  Detail & Related papers  (2025-09-01T09:44:01Z)
• Chain-of-Retrieval Augmented Generation [91.02950964802454]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.<n>Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval [108.9772640854136]
  Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via fine-tuning. We introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pre-training to accommodate the continuing of the corpus.
  arXiv  Detail & Related papers  (2024-07-16T08:42:36Z)
• Retrieving Continuous Time Event Sequences using Neural Temporal Point Processes with Learnable Hashing [24.963828650935913]
  We propose NeuroSeqRet, a first-of-its-kind framework designed specifically for end-to-end CTES retrieval. We develop four variants of the relevance model for different kinds of applications based on the trade-off between accuracy and efficiency. Our experiments show the significant accuracy boost of NeuroSeqRet as well as the efficacy of our hashing mechanism.
  arXiv  Detail & Related papers  (2023-07-13T18:54:50Z)
• Spatial-Temporal Graph Enhanced DETR Towards Multi-Frame 3D Object Detection [54.041049052843604]
  We present STEMD, a novel end-to-end framework that enhances the DETR-like paradigm for multi-frame 3D object detection. First, to model the inter-object spatial interaction and complex temporal dependencies, we introduce the spatial-temporal graph attention network. Finally, it poses a challenge for the network to distinguish between the positive query and other highly similar queries that are not the best match.
  arXiv  Detail & Related papers  (2023-07-01T13:53:14Z)
• Improving Out-of-Distribution Generalization of Neural Rerankers with Contextualized Late Interaction [52.63663547523033]
  Late interaction, the simplest form of multi-vector, is also helpful to neural rerankers that only use the [] vector to compute the similarity score. We show that the finding is consistent across different model sizes and first-stage retrievers of diverse natures.
  arXiv  Detail & Related papers  (2023-02-13T18:42:17Z)
• Improving Sequential Recommendations via Bidirectional Temporal Data Augmentation with Pre-training [46.5064172656298]
  We introduce Bidirectional temporal data Augmentation with pre-training (BARec)<n>Our approach leverages bidirectional temporal augmentation and knowledge-enhanced fine-tuning to synthesize authentic pseudo-prior items.<n>Our comprehensive experimental analysis on five benchmark datasets confirms the superiority of BARec across both short and elongated sequence contexts.
  arXiv  Detail & Related papers  (2021-12-13T07:33:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.