Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback

• URL: http://arxiv.org/abs/2606.00590v1
• Date: Sat, 30 May 2026 07:42:22 GMT
• Title: Critic-R: Improving Agentic Search using Instruction-tuned Retrievers with Natural Language Introspective Feedback
• Abstract summary: Critic-R is a framework that closes the feedback loop between the reasoning agent and the retrieval model during both inference and training.<n> Critic-R has two complementary mechanisms: Critic-R-Zero, an inference-time query refinement loop, and Critic-Embed, an optimization approach for retrieval models.<n>Results show that Critic-R significantly improves both retrieval quality and downstream answer accuracy.
• Score: 33.73030625999133
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic search systems iteratively interact with retrieval models to answer complex queries. Despite substantial progress, optimizing retrievers for agentic search remains challenging, often requiring heavy co-training or gold-standard annotations that limit real-world applicability. We propose Critic-R, a framework that explicitly closes the feedback loop between the reasoning agent and the retrieval model during both inference and training. Critic-R introduces a critic model that evaluates the agent's introspective reasoning trace after consuming retrieved evidence to determine whether the retrieved context sufficiently supports the next reasoning step. Critic-R has two complementary mechanisms: Critic-R-Zero, an inference-time query refinement loop that iteratively rewrites queries and retrieval instructions, and Critic-Embed, an optimization approach for retrieval models that leverages successful and failed refinement trajectories as automatic supervision without requiring manual relevance annotation. We evaluate Critic-R on HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle. Results show that Critic-R significantly improves both retrieval quality and downstream answer accuracy.

Related papers

• CAFE: Self-Improving Search Agents Need Co-Evolving Feedback [64.5257526832476]
  Outcome-supervised search agents learn when and how to retrieve evidence, but terminal rewards neither localize intermediate errors nor redirect an ongoing trajectory before those errors compound.<n>Treating corrective feedback as a learned in-trajectory intervention couples the two roles: the agent must decide when to request and use feedback, while the critic must infer useful corrections from outcome-confounded rollouts whose failure patterns shift as the agent improves.
  arXiv  Detail & Related papers  (2026-08-25T16:39:00Z)
• R$^2$-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Search [59.034878430183305]
  R$2$-Searcher explicitly explores and calibrates the retrieval and reasoning boundaries.<n>We show that R$2$-Searcher significantly outperforms state-of-the-art agentic search methods in answer accuracy and retrieval-reasoning quality.
  arXiv  Detail & Related papers  (2026-06-26T19:44:34Z)
• Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training [80.20022221643414]
  Cycle-Consistent Search is a gold-supervision-free framework for training search agents.<n>We show that CCS achieves performance comparable to supervised baselines.<n>These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.
  arXiv  Detail & Related papers  (2026-04-14T17:00:18Z)
• RASPRef: Retrieval-Augmented Self-Supervised Prompt Refinement for Large Reasoning Models [0.0]
  We introduce Retrieval-Augmented Self-Supervised Prompt Refinement (RASPRef), a framework that improves prompts without requiring human annotations or task-specific supervision.<n>RASPRef directly treats the prompt as the optimization target and improves it through an iterative retrieval-guided refinement process.
  arXiv  Detail & Related papers  (2026-03-27T21:49:21Z)
• CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic [24.371889836599138]
  CriticSearch is a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism.<n> Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines.
  arXiv  Detail & Related papers  (2025-11-15T11:06:57Z)
• RAVine: Reality-Aligned Evaluation for Agentic Search [7.4420114967110385]
  RAVine is a Reality-Aligned eValuation framework for agentic LLMs with search.<n> RAVine targets multi-point queries and long-form answers that better reflect user intents.<n>We benchmark a series of models using RAVine and derive several insights.
  arXiv  Detail & Related papers  (2025-07-22T16:08:12Z)
• Training Language Model to Critique for Better Refinement [58.73039433159486]
  We introduce textbfRefinement-oriented textbfCritique textbfOptimization (RCO), a novel framework designed to train critic models using refinement signals.<n>RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses.<n>By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment.
  arXiv  Detail & Related papers  (2025-06-27T12:10:57Z)
• ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning [48.01143057928348]
  We present ConvSearch-R1, a framework that eliminates dependency on external rewrite supervision by leveraging reinforcement learning to optimize reformulation directly through retrieval signals.<n>Our novel two-stage approach combines Self-Driven Policy Warm-Up to address the cold-start problem through retrieval-guided self-distillation, followed by Retrieval-Guided Reinforcement Learning with a specially designed rank-incentive reward shaping mechanism that addresses the sparsity issue in conventional retrieval metrics.
  arXiv  Detail & Related papers  (2025-05-21T17:27:42Z)
• Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimization [58.390885294401066]
  Retrieval-augmented generation (RAG) has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>We propose AlignRAG, a novel iterative framework grounded in Critique-Driven Alignment (CDA)<n>We introduce AlignRAG-auto, an autonomous variant that dynamically terminates refinement, removing the need to pre-specify the number of critique iterations.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks [68.49251303172674]
  State-of-the-art large language models (LLMs) exhibit impressive problem-solving capabilities but may struggle with complex reasoning and factual correctness. Existing methods harness the strengths of chain-of-thought and retrieval-augmented generation (RAG) to decompose a complex problem into simpler steps and apply retrieval to improve factual correctness. We introduce Critic-guided planning with Retrieval-augmentation, CR-Planner, a novel framework that leverages fine-tuned critic models to guide both reasoning and retrieval processes through planning.
  arXiv  Detail & Related papers  (2024-10-02T11:26:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.