Not All Queries Need Rewriting: When Prompt-Only LLM Refinement Helps and Hurts Dense Retrieval

• URL: http://arxiv.org/abs/2603.13301v1
• Date: Mon, 02 Mar 2026 10:40:10 GMT
• Title: Not All Queries Need Rewriting: When Prompt-Only LLM Refinement Helps and Hurts Dense Retrieval
• Abstract summary: We show that prompt-only rewriting can be harmful in well-optimized verticals.<n>We identify a consistent mechanism: rewritten terms co-occur with reduced lexical alignment.<n>We suggest that domain-adaptive post-training is a safer strategy when supervision or implicit feedback is available.
• Score: 0.11280931253550518
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt-only, single-step LLM query rewriting, where a rewrite is generated from the query alone without retrieval feedback, is commonly used in production RAG pipelines, but its effect on dense retrieval is poorly understood. We present a systematic empirical study across three BEIR benchmarks, two dense retrievers, and multiple training configurations, and find strongly domain-dependent behavior: rewriting degrades nDCG@10 by 9.0 percent on FiQA, improves it by 5.1 percent on TREC-COVID, and has no significant effect on SciFact. We identify a consistent mechanism: degradations co-occur with reduced lexical alignment between rewritten queries and relevant documents, as rewriting replaces domain-specific terms in already well-matched queries. In contrast, improvements arise when rewriting shifts queries toward corpus-preferred terminology and resolves inconsistent nomenclature. Lexical substitution occurs in 95 percent of rewrites across all outcome groups, showing that effectiveness depends on the direction of substitution rather than substitution itself. We also study selective rewriting and find that simple feature-based gating can reduce worst-case regressions but does not reliably outperform never rewriting, with even oracle selection offering only modest gains. Overall, these results show that prompt-only rewriting can be harmful in well-optimized verticals and suggest that domain-adaptive post-training is a safer strategy when supervision or implicit feedback is available.

Related papers

• SPEAR: Selection-aware Personalized End-to-end Adaptive Rewriting and Retrieval for Community Search [16.046037914956504]
  We propose SPEAR, which integrates three components that each target one failure mode.<n>The proposed framework improves rewrite semantic similarity@10 by +18.2 and click recall@10 by +99.5 over the production baseline.
  arXiv  Detail & Related papers  (2026-08-03T06:03:58Z)
• Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval [0.0]
  Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding their responses in external knowledge.<n>This paper presents an Agent-Orchestrated Adaptive RAG framework that introduces dynamic query decomposition, iterative retrieval, and a bounded self-reflective evaluation loop.
  arXiv  Detail & Related papers  (2026-06-04T03:38:46Z)
• Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning [47.963236269307735]
  Rerankers play a pivotal role in refining retrieval results for Retrieval-Augmented Generation.<n>Current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process.<n>We introduce ReRanking Preference Optimization (RRPO), a reinforcement learning framework that directly aligns reranking with the LLM's generation quality.
  arXiv  Detail & Related papers  (2026-04-02T14:19:47Z)
• Test-Time Strategies for More Efficient and Accurate Agentic RAG [58.44913384057518]
  Retrieval-Augmented Generation (RAG) systems face challenges with complex, multihop questions.<n>Such approaches can introduce inefficiencies, including repetitive retrieval of previously processed information.<n>We investigate test-time modifications to the Search-R1 pipeline to mitigate these identified shortcomings.
  arXiv  Detail & Related papers  (2026-03-12T19:18:59Z)
• Improving Neural Retrieval with Attribution-Guided Query Rewriting [3.1153758106426603]
  Underspecified or ambiguous queries can misdirect ranking even when relevant documents exist.<n>We propose an attribution-guided query rewriting method that uses token-level explanations to guide rewriting.<n>The resulting rewrites consistently improve retrieval effectiveness over strong baselines.
  arXiv  Detail & Related papers  (2026-02-12T11:34:06Z)
• Relevance to Utility: Process-Supervised Rewrite for RAG [38.81331265140413]
  We show how "bridge" modules fail to capture true document utility.<n>We propose R2U, with a key distinction of directly optimizing to maximize the probability of generating a correct answer.
  arXiv  Detail & Related papers  (2025-09-19T04:24:57Z)
• QueryBandits for Hallucination Mitigation: Exploiting Semantic Features for No-Regret Rewriting [27.117502300924183]
  We introduce QueryBandits, a framework that designs rewrite strategies to maximize a reward model.<n>We empirically substantiate the effectiveness of QueryBandits in mitigating hallucination via the intervention that takes the form of a query rewrite.
  arXiv  Detail & Related papers  (2025-08-22T01:41:49Z)
• Gumbel Reranking: Differentiable End-to-End Reranker Optimization [61.16471123356738]
  RAG systems rely on rerankers to identify relevant documents.<n> fine-tuning these models remains challenging due to the scarcity of annotated query-document pairs.<n>We propose Gumbel Reranking, an end-to-end training framework for rerankers aimed at minimizing the training-inference gap.
  arXiv  Detail & Related papers  (2025-02-16T13:23:39Z)
• Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers [66.55612528039894]
  AdaQR is a framework for training query rewriting models with limited rewrite annotations from seed datasets and completely no passage label. A novel approach is proposed to assess retriever's preference for these candidates by the probability of answers conditioned on the conversational query.
  arXiv  Detail & Related papers  (2024-06-16T16:09:05Z)
• Generative Query Reformulation Using Ensemble Prompting, Document Fusion, and Relevance Feedback [8.661419320202787]
  GenQREnsemble and GenQRFusion leverage paraphrases of a zero-shot instruction to generate multiple sets of keywords to improve retrieval performance. We demonstrate that an ensemble of query reformulations can improve retrieval effectiveness by up to 18% on nDCG@10 in pre-retrieval settings and 9% on post-retrieval settings.
  arXiv  Detail & Related papers  (2024-05-27T21:03:26Z)
• RaFe: Ranking Feedback Improves Query Rewriting for RAG [83.24385658573198]
  We propose a framework for training query rewriting models free of annotations. By leveraging a publicly available reranker, oursprovides feedback aligned well with the rewriting objectives.
  arXiv  Detail & Related papers  (2024-05-23T11:00:19Z)
• LoL: A Comparative Regularization Loss over Query Reformulation Losses for Pseudo-Relevance Feedback [70.44530794897861]
  Pseudo-relevance feedback (PRF) has proven to be an effective query reformulation technique to improve retrieval accuracy. Existing PRF methods independently treat revised queries originating from the same query but using different numbers of feedback documents. We propose the Loss-over-Loss (LoL) framework to compare the reformulation losses between different revisions of the same query during training.
  arXiv  Detail & Related papers  (2022-04-25T10:42:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.