Prompt-Level Reward Specifications for Open-Ended Post-Training

• URL: http://arxiv.org/abs/2605.29275v1
• Date: Thu, 28 May 2026 02:52:06 GMT
• Title: Prompt-Level Reward Specifications for Open-Ended Post-Training
• Abstract summary: Open-ended post-training benefits from rewards that make prompt-specific success conditions explicit.<n>We propose a prompt-level reward specification framework that separates reward specification from reward.
• Score: 43.215398565987265
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-ended post-training benefits from rewards that make prompt-specific success conditions explicit, rather than relying only on post-hoc scalar scores. In instruction following, writing, and decision-support tasks, response quality depends on local requirements, holistic preferences, and explicit constraints, but existing reward methods often leave these criteria implicit or cover only narrowly verifiable cases. We propose a prompt-level reward specification framework that separates reward specification from reward computation. Given only prompts, our framework constructs reusable task-adaptive rubrics and executable hard-constraint checkers offline, making reward criteria explicit before training and reusable across rollouts. At scoring time, artifact-anchored rubric and code scores are combined with an independent global score for residual holistic quality, yielding a normalized hybrid reward over requirement satisfaction, holistic quality, and deterministic constraints. The framework requires no human preference annotations, reference answers, or a separately trained reward model. Experiments show that the resulting reward improves offline RM-style response ranking and supports online reinforcement learning across multiple open-ended benchmarks. Ablations further show that rubrics, global scoring, and executable verification provide complementary supervision.

Related papers

• Claim-Level Rubric Rewards for Video Caption Reinforcement Learning [44.96572145476932]
  Claim-Level Rewards Rewards (CuRe) is a structured reward framework designed to address the reward-design bottleneck in reinforcement learning for dense video captioning.<n>CuRe reformulates reward modeling as fine-grained claim-level verification.
  arXiv  Detail & Related papers  (2026-07-06T14:33:41Z)
• Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search [8.82040111147814]
  We present an end-to-end RLAIF framework to generate emphportable search queries.<n>This task introduces a highly adversarial reward surface where policy optimization exploits flaws in rubrics, resulting in verbatim-copying behaviors.<n>We show that introducing a rule-based reward floor to correct for rewards assigned to verbatim copying degenerates this failure mode, resulting in a substantial $+0.147$ quality improvement on a cross-family evaluation judge.
  arXiv  Detail & Related papers  (2026-06-25T17:09:12Z)
• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill [36.002795736704]
  We propose Skill Reward Model (Skill-RM), a unified framework that reformulates reward modeling as the execution of a reusable Reward-Evaluation Skill.<n>By treating reward computation as a structured agentic task, Skill-RM provides a consistent interface to orchestrate heterogeneous resources.<n>Our findings suggest that Skill-RM not only provides a unified solution for reward modeling but also achieves superior performance through the strategic and dynamic orchestration of evidence.
  arXiv  Detail & Related papers  (2026-06-02T17:56:57Z)
• QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards [68.15832368034815]
  We present a framework that co-designs queries and rubrics.<n>QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline.
  arXiv  Detail & Related papers  (2026-06-02T17:53:04Z)
• RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses [7.123785374544969]
  Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives.<n>We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy.<n>We propose textscRHyVE, a competence-aware verification and phase-aware deployment protocol.
  arXiv  Detail & Related papers  (2026-04-30T16:01:51Z)
• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning [52.48243762705385]
  Chain of Uncertain Rewards (CoUR) is a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation.<n>We show that CoUR achieves better performance and significantly lowers the cost of reward evaluations.
  arXiv  Detail & Related papers  (2026-04-15T05:44:14Z)
• Likelihood-Based Reward Designs for General LLM Reasoning [17.640897774014707]
  Fine-tuning large language models via reinforcement learning requires a specific reward function, often binary, for each benchmark.<n>Here, we investigate rewards derived from the probability or log-probability of emitting the reference answer.<n>We find that using the log-probability of the reference answer as the reward for chain-of-thought learning is the only option that performs well in all setups.
  arXiv  Detail & Related papers  (2026-02-03T20:04:21Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewards [11.149294285483782]
  We propose a unified RLVR-based training paradigm that bridges the gap between non-verifiable tasks and verifiable rewards.<n>We introduce a writing-principle-based pairwise Generative Reward Model (GenRM) and a novel Bootstrapped Relative Policy Optimization (BRPO) algorithm.<n>Our approach empowers LLMs to develop robust writing capabilities without supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-05-30T14:34:57Z)
• Reward Collapse in Aligning Large Language Models [73.78408135192471]
  We study the phenomenon of textitreward collapse', an empirical observation where the prevailing ranking-based approach results in an textitidentical reward distribution.<n>Our experimental results suggest that our proposed prompt-aware utility functions significantly alleviate reward collapse during the training of reward models.
  arXiv  Detail & Related papers  (2023-05-28T02:12:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.