Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search

• URL: http://arxiv.org/abs/2606.27291v1
• Date: Thu, 25 Jun 2026 17:09:12 GMT
• Title: Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search
• Abstract summary: We present an end-to-end RLAIF framework to generate emphportable search queries.<n>This task introduces a highly adversarial reward surface where policy optimization exploits flaws in rubrics, resulting in verbatim-copying behaviors.<n>We show that introducing a rule-based reward floor to correct for rewards assigned to verbatim copying degenerates this failure mode, resulting in a substantial $+0.147$ quality improvement on a cross-family evaluation judge.
• Score: 8.82040111147814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Job-search platforms rely on low-bandwidth query interfaces that often fail to capture the high-dimensional complexity of candidate profiles. We present an end-to-end RLAIF (Reinforcement Learning from AI Feedback) framework to generate \emph{portable} job search queries, terms that abstract away seeker-specific identifiers while preserving generalizable qualifications. This task introduces a highly adversarial reward surface where policy optimization frequently exploits flaws in LLM-as-judge rubrics, resulting in degenerate verbatim-copying behaviors. We conducted comprehensive empirical experiments to isolate the impact of optimization mechanics against structured reward engineering. Our results demonstrate that for critic-free optimizers, performance is overwhelmingly dictated by robust reward shaping, rendering the specific choice of algorithm largely immaterial. While critic-free per-rollout baseline methods (RLOO and REINFORCE++) natively resist reward-hacking, the group-relative advantage normalization in GRPO appears uniquely sensitive to spurious reward signals, making it disproportionately susceptible to exploitation. We show that introducing a deterministic, rule-based reward floor to correct for rewards assigned to verbatim copying mitigates this failure mode, resulting in a substantial $+0.147$ quality improvement on a cross-family evaluation judge. Ultimately, we show that the training-time reward model inflates performance gains by $2.4\times$, confirming that the training success is fundamentally dependent on enforcing reward-shaping disciplines rather than selecting alternative optimizers.

Related papers

• Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning [18.91502176461499]
  Machine unlearning seeks to remove specific knowledge from trained language models without full retraining.<n>Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem.<n>We study how reward design affects unlearning efficiency within the Reinforcement Unlearning framework.
  arXiv  Detail & Related papers  (2026-07-30T10:15:59Z)
• RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents [24.215311212134882]
  We propose Reward-Swap Policy Optimization (RSPO) to leverage the rich information from dense process rewards to facilitate training with outcome rewards.<n>RSPO ensures the diversity of sampled trajectories while guaranteeing consistency between the optimization objective and the true outcome rewards.
  arXiv  Detail & Related papers  (2026-07-06T06:32:39Z)
• When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study [77.64957118012423]
  Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning.<n>Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores.<n>We present a systematic empirical study of intrinsic reward methods for code generation.
  arXiv  Detail & Related papers  (2026-06-18T19:15:50Z)
• Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning [52.48243762705385]
  Chain of Uncertain Rewards (CoUR) is a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation.<n>We show that CoUR achieves better performance and significantly lowers the cost of reward evaluations.
  arXiv  Detail & Related papers  (2026-04-15T05:44:14Z)
• RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time [51.256131853751754]
  We show that teaching reward models to produce explicit, multi-dimensional critiques transforms them from passive evaluators into active optimization tools.<n>We introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data.<n>The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models.
  arXiv  Detail & Related papers  (2026-04-13T15:38:09Z)
• GDRO: Group-level Reward Post-training Suitable for Diffusion Models [55.948229011478304]
  Group-level rewards successfully align the model with the targeted reward.<n>Group-level Direct Reward Optimization (GDRO) is a new post-training paradigm for group-level reward alignment.<n>GDRO supports full offline training that saves the large time cost for image rollout sampling.<n>It is diffusion-sampler-independent, which eliminates the need for the ODE-to-SDE approximation to obtainity.
  arXiv  Detail & Related papers  (2026-01-05T11:47:18Z)
• PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning [29.373324685358753]
  We propose PROF, a framework to generate and improve executable reward function codes from natural language descriptions and a single expert trajectory.<n>We also propose Reward Preference Ranking (RPR), a novel reward function quality assessment and ranking strategy without requiring environment interactions or RL training.
  arXiv  Detail & Related papers  (2025-11-14T14:38:02Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment [54.787826863212146]
  Inference-time computation offers a powerful axis for scaling the performance of language models.<n>We analyze the performance of inference-time alignment algorithms in terms of (i) response quality, and (ii) compute.<n>We introduce $textttInferenceTimePessimism$, a new algorithm which mitigates reward hacking through deliberate use of inference-time compute.
  arXiv  Detail & Related papers  (2025-03-27T18:00:08Z)
• Behavior Alignment via Reward Function Optimization [23.92721220310242]
  We introduce a new framework that integrates auxiliary rewards reflecting a designer's domain knowledge with the environment's primary rewards. We evaluate our method's efficacy on a diverse set of tasks, from small-scale experiments to high-dimensional control challenges.
  arXiv  Detail & Related papers  (2023-10-29T13:45:07Z)
• Mind the Gap: Offline Policy Optimization for Imperfect Rewards [14.874900923808408]
  We propose a unified offline policy optimization approach, textitRGM (Reward Gap Minimization), which can handle diverse types of imperfect rewards. By exploiting the duality of the lower layer, we derive a tractable algorithm that enables sampled-based learning without any online interactions.
  arXiv  Detail & Related papers  (2023-02-03T11:39:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.