Reward-Free Evolving Agents via Pairwise Validator

• URL: http://arxiv.org/abs/2607.14408v1
• Date: Wed, 15 Jul 2026 22:50:32 GMT
• Title: Reward-Free Evolving Agents via Pairwise Validator
• Abstract summary: A self-evolving agentic loop repeatedly proposes a tweaked version of an agent.<n>It accepts or rejects the change based on a per-iteration quality signal.<n>We propose replacing the scalar at the accept/reject gate with a pairwise validator.
• Score: 12.6053820984668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A self-evolving agentic loop repeatedly proposes a tweaked version of an agent (its prompt template or program) and accepts or rejects the change based on a per-iteration quality signal. Designing that signal is often the costly part of the project: a reliable scalar reward requires domain expertise and labeled examples that are themselves as expensive to assemble as the agent's underlying task. We propose replacing the scalar at the accept/reject gate with a pairwise validator: a frozen LLM that, given the parent and child candidate, returns a binary verdict on which is better. Pairwise judgment is generally easier and more stable than absolute scoring, due to its contrastive nature, which mitigates the need for strict scale calibration. The validator also requires no training of its own. We integrate the validator into three published self-evolving engines (GEPA, ADRS, ShinkaEvolve) and report two flavors: Adaptive Focus, which retains the engine's existing val-set parent selection, and Soft Elo, which lets the validator's verdicts drive parent selection so that val-set rewards drop as well. Across multiple agents and two artifact substrates (prompt and code), our method matches or exceeds the full-reward baseline on the majority of settings we evaluate, and the pattern survives a cross-family validator swap. The pairwise gate is thus a drop-in replacement for per-step reward design at competitive task accuracy without the labeling cost.

Related papers

• HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution [11.431479238875712]
  We introduce HarnessEvolve, a self-evolving framework that learns from reference trajectories to achieve reliable agent self-evolution.<n>We conduct extensive experiments on several benchmarks spanning open-domain and enterprise scenarios, using different models and agent frameworks.<n>Results demonstrate that HarnessEvolve consistently outperforms state-of-the-art baselines across all benchmarks and settings.
  arXiv  Detail & Related papers  (2026-09-01T07:31:18Z)
• Shared Prefixes, Better Credit: Adaptive Routing for Multi-Agent Reasoning [72.49938949082117]
  Multi-agent reasoning (MAR) improves reasoning reliability through iterative solution exchange and refinement.<n>Existing adaptive MAR methods typically learn routing decisions from query-level labels or trajectory-level returns.<n>We propose TreeCredit, a shared-operator credit assignment framework for efficient adaptive MAR.
  arXiv  Detail & Related papers  (2026-08-03T14:22:23Z)
• Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA [0.0]
  We introduce CARE (Confidence-Adaptive Routing of Experts), which admits experts in a nucleus fashion.<n> CARE improves over fixed top-k MoE-LoRA at matched compute and matches the fixed-k=4 baseline while activating fewer experts.<n>The same confidence and disagreement signals also improve out-of-distribution detection over MSP, entropy, and multi-pass proxies.
  arXiv  Detail & Related papers  (2026-07-28T17:59:16Z)
• AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows [1.6785141970297952]
  AEVAL (Agentic Evaluation) is a CI-integrated framework that replaces this practice with a deterministic, reproducible test pipeline for agentic skills.<n>Every skill change triggers a test event: the skill runs against a developer declared evaluation contract inside an automated executor.<n>A key ingredient is a structural separation between executor and grader, preventing a subtle but pervasive failure mode.
  arXiv  Detail & Related papers  (2026-07-16T21:33:05Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Token-Level Generalization in LoRA Adapter Backdoors: Attack Characterization and Behavioral Detection [0.0]
  We show that LoRA can be reliably backdoored through training data poisoning.<n>The attack scales monotonically with rank, and the chosen trigger-anchor token is both trigger-dependent and base-model-dependent.<n> Behavioral detection is the operationally portable result for supply chain scanning.
  arXiv  Detail & Related papers  (2026-05-28T16:32:25Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL [76.45061154544568]
  Self-play reinforcement learning trains language models on their own generated tasks, co-evolving a proposer and solver without human labels.<n>Recent systems report strong reasoning gains, but collapse and instability are widely observed and poorly understood.<n>We argue instead that self-play stability is governed by two distinct levers: a data-level gate that decides which proposer-generated tasks enter the training pool, and the reward signal that updates the policy on tasks already admitted.
  arXiv  Detail & Related papers  (2026-05-21T09:19:23Z)
• Hybrid Reinforcement: When Reward Is Sparse, It's Better to Be Dense [36.71358559780692]
  HERO is a reinforcement learning framework that integrates verifier signals with reward-model scores in a structured way.<n> HERO consistently outperforms RM-only and verifier-only baselines, with strong gains on both verifiable and hard-to-verify tasks.
  arXiv  Detail & Related papers  (2025-10-08T17:09:41Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.