EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning

• URL: http://arxiv.org/abs/2609.12459v1
• Date: Fri, 11 Sep 2026 05:40:59 GMT
• Title: EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
• Abstract summary: EvoRS is a self-evolving RL framework that evolves the reward system from on-policy experience.<n>Across writing and roleplay, EvoRS achieves the best quality under all three judges.
• Score: 54.99968863245366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by \(2.107\) and \(4.767\) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.

Related papers

• Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning [18.91502176461499]
  Machine unlearning seeks to remove specific knowledge from trained language models without full retraining.<n>Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem.<n>We study how reward design affects unlearning efficiency within the Reinforcement Unlearning framework.
  arXiv  Detail & Related papers  (2026-07-30T10:15:59Z)
• CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning [6.874780979291594]
  CoRe is a framework that integrates Formal Rewards (FR) and Residual Rewards (RR) with vision-language models (VLMs) feedback to achieve preference-aligned policies.<n>Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capture nuanced rewards that complement FR, ensuring alignment with human intent.
  arXiv  Detail & Related papers  (2026-07-02T05:20:47Z)
• Learning Process Rewards via Success Visitation Matching for Efficient RL [54.67547527674048]
  We propose a simple approach to transform a sparse outcome reward into a dense process reward.<n>Our approach relies on training a discriminator to distinguish between previous successful and unsuccessful episodes.<n>By incentivizing the policy to match the visitations over all states, not just those that correspond to task success, this reward provides dense feedback on whether progress is being made towards task completion.
  arXiv  Detail & Related papers  (2026-06-22T17:30:24Z)
• When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study [77.64957118012423]
  Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning.<n>Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores.<n>We present a systematic empirical study of intrinsic reward methods for code generation.
  arXiv  Detail & Related papers  (2026-06-18T19:15:50Z)
• Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning [18.80588864499134]
  Grad2Reward extracts dense process rewards directly from the Judge's model inference process via a single backward pass.<n>By leveraging gradient-based attribution, Grad2Reward enables precise token-level credit assignment.<n>Experiments demonstrate that optimized policies with Grad2Reward achieve outstanding performance across diverse open-ended tasks.
  arXiv  Detail & Related papers  (2026-02-02T08:13:13Z)
• Stabilizing Long-term Multi-turn Reinforcement Learning with Gated Rewards [13.70228195630989]
  We introduce the SWE-oriented RL Framework, a unified system supporting multi-turn interaction, docker-based execution, and customizable reward functions.<n>We also propose Gated Reward Accumulation (G-RA), a novel method that accumulates immediate rewards only when high-level (long-term) rewards meet a predefined threshold.
  arXiv  Detail & Related papers  (2025-08-14T11:37:02Z)
• Adversarial Training of Reward Models [74.17196154247964]
  We introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples.<n>By leveraging reinforcement learning, Adv-RM trains a policy to expose vulnerabilities in large state-of-the-art reward models.<n>We demonstrate that Adv-RM significantly outperforms conventional reward training.
  arXiv  Detail & Related papers  (2025-04-08T15:38:25Z)
• Process Reinforcement through Implicit Rewards [94.09453548052862]
  Dense process rewards have proven a more effective alternative to the sparse outcome-level rewards in the inference-time scaling of large language models (LLMs)<n>Dense rewards also offer an appealing choice for the reinforcement learning (RL) of LLMs since their fine-grained rewards have the potential to address some inherent issues of outcome rewards.<n>This can be primarily attributed to the challenges of training process reward models (PRMs) online, where collecting high-quality process labels is prohibitively expensive.<n>We propose PRIME, which enables online PRM updates using only policy rollouts and outcome labels through implict process rewards
  arXiv  Detail & Related papers  (2025-02-03T15:43:48Z)
• Deceptive Sequential Decision-Making via Regularized Policy Optimization [54.38738815697299]
  We model autonomous systems as Markov decision processes, with adversaries using inverse reinforcement learning to recover reward functions.<n>We show three regularization strategies for policy synthesis problems that actively deceive an adversary about a system's reward.<n>We show that diversionary, targeted, and equivocal deception all steer the adversary to false beliefs while still attaining a total accumulated reward that is at least 97% of its optimal, non-deceptive value.
  arXiv  Detail & Related papers  (2025-01-30T23:41:40Z)
• ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization [41.074747242532695]
  Online Reward Selection and Policy Optimization (ORSO) is a novel approach that frames the selection of shaping reward function as an online model selection problem.<n>ORSO significantly reduces the amount of data required to evaluate a shaping reward function, resulting in superior data efficiency and a significant reduction in computational time (up to 8 times)<n>ORSO consistently identifies high-quality reward functions outperforming prior methods by more than 50% and on average identifies policies as performant as the ones learned using manually engineered reward functions by domain experts.
  arXiv  Detail & Related papers  (2024-10-17T17:55:05Z)
• Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning [19.788336796981685]
  We propose a novel Distributional Reward Estimation framework for effective Multi-Agent Reinforcement Learning (DRE-MARL) Our main idea is to design the multi-action-branch reward estimation and policy-weighted reward aggregation for stabilized training. The superiority of the DRE-MARL is demonstrated using benchmark multi-agent scenarios, compared with the SOTA baselines in terms of both effectiveness and robustness.
  arXiv  Detail & Related papers  (2022-10-14T08:31:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.