RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS

• URL: http://arxiv.org/abs/2512.04552v1
• Date: Thu, 04 Dec 2025 08:12:49 GMT
• Title: RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS
• Abstract summary: Policy models can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading quality.<n>We propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme.<n>This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions.
• Score: 37.07631000345777
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentiable reinforcement learning (RL) frameworks like DiffRO offer a powerful approach for controllable text-to-speech (TTS), but are vulnerable to reward hacking, particularly for nuanced tasks like emotion control. The policy model can exploit a vanilla Reward Model (RM) by generating acoustic artifacts to achieve spurious rewards, but at the cost of degrading perceptual quality. To address this, we propose Robust Reward Policy Optimization (RRPO), a novel framework that employs a hybrid regularization scheme. This scheme develops a robust RM whose reward signal is more reliably aligned with human perception, compelling the policy to abandon detrimental shortcuts and instead learn the complex features of genuine emotions. Our ablation study confirms the enhanced robustness of our RM, as evidenced by its strong cross-lingual generalization. The subjective evaluation demonstrates that this robust RM effectively mitigates reward hacking, leading to significant improvements in both emotional expressiveness and naturalness over all baselines. Demo page: https://lrwinr.github.io/RRPO-CosyVoice.

Related papers

• The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement [32.29319213544292]
  Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data.<n>We propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training.
  arXiv  Detail & Related papers  (2026-05-29T06:25:16Z)
• ARES: Adaptive Red-Teaming and End-to-End Repair of Policy-Reward System [70.10833169681239]
  Reinforcement Learning from Human Feedback (RLHF) is central to aligning Large Language Models (LLMs)<n>It introduces a critical vulnerability: an imperfect Reward Model (RM) can become a single point of failure when it fails to penalize unsafe behaviors.<n>We present ARES, a framework that systematically discovers and mitigates such dual vulnerabilities.
  arXiv  Detail & Related papers  (2026-04-20T19:54:47Z)
• Mitigating Reward Hacking in RLHF via Advantage Sign Robustness [40.66292181512935]
  We make the assumption that reward hacking is often caused by flipped advantage signs.<n>By considering an adversarial perturbation in the RM parameter space, we can derive a certified sign-preservation radius.<n>We propose Sign-Certified Policy Optimization (SignCert-PO), down-weighting non-robust completions in the policy gradient update.
  arXiv  Detail & Related papers  (2026-04-03T11:45:16Z)
• R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging [69.96389360650072]
  We show that reasoning fidelity is highly predictive of downstream RLHF outcomes, beyond standard label accuracy.<n>We propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment.
  arXiv  Detail & Related papers  (2026-02-06T15:17:11Z)
• SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models [53.19726629537694]
  Post-training alignment of video generation models with human preferences is a critical goal.<n>Current data collection paradigms, reliant on in-prompt pairwise annotations, suffer from labeling noise.<n>We propose SoliReward, a systematic framework for video RM training.
  arXiv  Detail & Related papers  (2025-12-17T14:28:23Z)
• Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm [24.126318783226598]
  Inverse Reinforcement Learning (IRL) learns a reward function to explain expert demonstrations.<n>Modern IRL methods often use the adversarial (minimax) formulation that alternates between reward and policy optimization.<n>Recent non-adversarial IRL approaches improve stability by jointly learning reward and policy via energy-based formulations.
  arXiv  Detail & Related papers  (2025-09-27T05:36:13Z)
• Energy-Based Reward Models for Robust Language Model Alignment [9.843359827321194]
  We introduce Energy-Based Reward Model (EBRM), a lightweight post-hoc refinement framework for Reward Models (RMs)<n>EBRM models the reward distribution explicitly, capturing uncertainty in human preferences and mitigating the impact of noisy or misaligned annotations.<n> Empirical evaluations demonstrate significant improvements in robustness and generalization, achieving up to a 5.97% improvement in safety-critical alignment tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:47:15Z)
• Adversarial Training of Reward Models [74.17196154247964]
  We introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples.<n>By leveraging reinforcement learning, Adv-RM trains a policy to expose vulnerabilities in large state-of-the-art reward models.<n>We demonstrate that Adv-RM significantly outperforms conventional reward training.
  arXiv  Detail & Related papers  (2025-04-08T15:38:25Z)
• RRM: Robust Reward Model Training Mitigates Reward Hacking [51.12341734942797]
  Reward models (RMs) play a pivotal role in aligning large language models with human preferences.<n>We introduce a causal framework that learns preferences independent of these artifacts.<n>Experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model.
  arXiv  Detail & Related papers  (2024-09-20T01:46:07Z)
• Confronting Reward Model Overoptimization with Constrained RLHF [114.71591361764547]
  We show that correlation between component RMs has a significant effect on the locations of these points. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers.
  arXiv  Detail & Related papers  (2023-10-06T16:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.