Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training

• URL: http://arxiv.org/abs/2604.12967v1
• Date: Tue, 14 Apr 2026 17:00:18 GMT
• Title: Cycle-Consistent Search: Question Reconstructability as a Proxy Reward for Search Agent Training
• Abstract summary: Cycle-Consistent Search is a gold-supervision-free framework for training search agents.<n>We show that CCS achieves performance comparable to supervised baselines.<n>These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.
• Score: 80.20022221643414
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) has shown strong potential for optimizing search agents in complex information retrieval tasks. However, existing approaches predominantly rely on gold supervision, such as ground-truth answers, which is difficult to scale. To address this limitation, we propose Cycle-Consistent Search (CCS), a gold-supervision-free framework for training search agents, inspired by cycle-consistency techniques from unsupervised machine translation and image-to-image translation. Our key hypothesis is that an optimal search trajectory, unlike insufficient or irrelevant ones, serves as a lossless encoding of the question's intent. Consequently, a high-quality trajectory should preserve the information required to accurately reconstruct the original question, thereby inducing a reward signal for policy optimization. However, naive cycle-consistency objectives are vulnerable to information leakage, as reconstruction may rely on superficial lexical cues rather than the underlying search process. To reduce this effect, we apply information bottlenecks, including exclusion of the final response and named entity recognition (NER) masking of search queries. These constraints force reconstruction to rely on retrieved observations together with the structural scaffold, ensuring that the resulting reward signal reflects informational adequacy rather than linguistic redundancy. Experiments on question-answering benchmarks show that CCS achieves performance comparable to supervised baselines while outperforming prior methods that do not rely on gold supervision. These results suggest that CCS provides a scalable training paradigm for training search agents in settings where gold supervision is unavailable.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent [16.60550424408283]
  Existing search-augmented LLM agents are trained using Reinforcement Learning to boost its reasoning capabilities.<n>We propose PROGRESS which utilizes teacher-guided coverage reward to explicitly shape decomposed query generation.<n> integrated into an R1-style training framework, our approach provides lightweight guidance over query decomposition decisions.
  arXiv  Detail & Related papers  (2026-08-02T03:48:41Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing [12.09179106162719]
  We propose Skill-RAG, a failure-aware RAG framework that couples a hidden-state prober with a prompt-based skill router.<n> Experiments show that Skill-RAG substantially improves accuracy on hard cases persisting after multi-turn retrieval.
  arXiv  Detail & Related papers  (2026-04-17T07:25:43Z)
• SIGHT: Reinforcement Learning with Self-Evidence and Information-Gain Diverse Branching for Search Agent [39.43590030917357]
  SIGHT is a framework that enhances search-based reasoning through Self-Evidence Support and Information-Gain Driven Diverse Branching.<n>SIGHT distills search results into high-fidelity evidence via SES and calculates an Information Gain score to pinpoint pivotal states.<n> Experiments on single-hop and multi-hop QA benchmarks demonstrate that SIGHT significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2026-02-12T04:16:55Z)
• Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration [49.9937230730202]
  We propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention.<n>Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories.<n>We show that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales.
  arXiv  Detail & Related papers  (2026-02-03T15:32:09Z)
• RE-TRAC: REcursive TRAjectory Compression for Deep Search Agents [144.5598958575922]
  Re-TRAC is an agentic framework that performs cross-trajectory exploration.<n>We show that Re-TRAC consistently outperforms ReAct by 15-20% on BrowseComp with frontier LLMs.
  arXiv  Detail & Related papers  (2026-02-02T18:58:07Z)
• Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [60.0970117192627]
  Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents.<n>Existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process.<n>We propose textbfCitation-aware RL Rewards (CaRR), a fine-grained reward framework for deep search agents.
  arXiv  Detail & Related papers  (2026-01-09T18:57:53Z)
• Beyond Monolithic Architectures: A Multi-Agent Search and Knowledge Optimization Framework for Agentic Search [56.78490647843876]
  Agentic search has emerged as a promising paradigm for complex information seeking by enabling Large Language Models (LLMs) to interleave reasoning with tool use.<n>We propose bfM-ASK, a framework that explicitly decouples agentic search into two complementary roles: Search Behavior Agents, which plan and execute search actions, and Knowledge Management Agents, which aggregate, filter, and maintain a compact internal context.
  arXiv  Detail & Related papers  (2026-01-08T08:13:27Z)
• CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic [24.371889836599138]
  CriticSearch is a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism.<n> Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines.
  arXiv  Detail & Related papers  (2025-11-15T11:06:57Z)
• Repurposing Synthetic Data for Fine-grained Search Agent Supervision [81.95597592711688]
  LLM-based search agents are increasingly trained on entity-centric synthetic data.<n> prevailing training methods discard this rich entity information, relying instead on sparse, outcome-based rewards.<n>We introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function.
  arXiv  Detail & Related papers  (2025-10-28T17:50:40Z)
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards [18.92867715736209]
  We propose ReSeek, a novel self-correcting framework for training search agents.<n>Our framework introduces a self-correction mechanism that empowers the agent to dynamically identify and recover from erroneous search paths.<n>To mitigate the risk of data contamination in existing datasets, we introduce FictionalHot.
  arXiv  Detail & Related papers  (2025-10-01T06:44:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.