RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time

• URL: http://arxiv.org/abs/2604.11626v2
• Date: Tue, 14 Apr 2026 06:06:15 GMT
• Title: RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time
• Abstract summary: We show that teaching reward models to produce explicit, multi-dimensional critiques transforms them from passive evaluators into active optimization tools.<n>We introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data.<n>The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models.
• Score: 51.256131853751754
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most reward models for visual generation reduce rich human judgments to a single unexplained score, discarding the reasoning that underlies preference. We show that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, improving generators in two complementary ways: at training time, structured rationales provide interpretable, fine-grained rewards for reinforcement learning; at test time, a Generate-Critique-Refine loop turns critiques into targeted prompt revisions that improve outputs without any parameter updates. To train such a reward model without costly rationale annotations, we introduce Preference-Anchored Rationalization (PARROT), a principled framework that recovers high-quality rationales from readily available preference data through anchored generation, consistency filtering, and distillation. The resulting model, RationalRewards (8B), achieves state-of-the-art preference prediction among open-source reward models, competitive with Gemini-2.5-Pro, while using 10-20x less training data than comparable baselines. As an RL reward, it consistently improves text-to-image and image-editing generators beyond scalar alternatives. Most strikingly, its test-time critique-and-refine loop matches or exceeds RL-based fine-tuning on several benchmarks, suggesting that structured reasoning can unlock latent capabilities in existing generators that suboptimal prompts fail to elicit.

Related papers

• Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling [39.325422496065435]
  Multi-Role Generation (MRRG) is a training-free and reference-free framework that elicits evaluation criteria from multiple roles.<n>MRRG can be used both to validate pairwise preferences and to provide rewards for RLO-style Reinforcement Learning with Verifiable Rewards (RLVR)
  arXiv  Detail & Related papers  (2026-07-02T07:50:38Z)
• Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling [32.50799160769865]
  Video reward models should predict accurate rewards that align with human preferences across diverse scenarios.<n>Existing paradigms face a fundamental dilemma: textitDiscriminative RMs regress rewards directly on features extracted by multimodal large language models without explicit reasoning.<n>We introduce DeScore, a training-efficient and generalizable video reward model.
  arXiv  Detail & Related papers  (2026-05-07T09:30:58Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• Alternating Reinforcement Learning with Contextual Rubric Rewards [2.1172256884504588]
  Reinforcement Learning with Rewards (RLRR) is a framework that extends conventional reinforcement learning from human feedback.<n>Existing approaches in RLRR are limited to linearly compressing vector rewards into a scalar reward with a fixed weightings.<n>This work proposes Alternating Reinforcement Learning with Rewards (ARL-RR) to overcome the limitations of reward aggregation.
  arXiv  Detail & Related papers  (2026-03-04T04:18:39Z)
• R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judging [69.96389360650072]
  We show that reasoning fidelity is highly predictive of downstream RLHF outcomes, beyond standard label accuracy.<n>We propose Rationale-Centric Alignment, R-Align, which augments training with gold judgments and explicitly supervises rationale alignment.
  arXiv  Detail & Related papers  (2026-02-06T15:17:11Z)
• GDRO: Group-level Reward Post-training Suitable for Diffusion Models [55.948229011478304]
  Group-level rewards successfully align the model with the targeted reward.<n>Group-level Direct Reward Optimization (GDRO) is a new post-training paradigm for group-level reward alignment.<n>GDRO supports full offline training that saves the large time cost for image rollout sampling.<n>It is diffusion-sampler-independent, which eliminates the need for the ODE-to-SDE approximation to obtainity.
  arXiv  Detail & Related papers  (2026-01-05T11:47:18Z)
• Reinforced Preference Optimization for Recommendation [28.87206911186567]
  We propose Reinforced Preference Optimization for Recommendation (ReRe) for generative recommenders.<n>ReRe incorporates constrained beam search to improve sampling efficiency and diversify hard negatives.<n>We show that ReRe consistently outperforms both traditional and LLM-based recommenders in ranking performance.
  arXiv  Detail & Related papers  (2025-10-14T07:04:33Z)
• OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment [38.1645520104553]
  We introduce OpenRubrics, a large-scale collection of (prompt,explicit) pairs for training rubric-generation and rubric-based reward models.<n>To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Generation (CRG), which derives both hard rules (implicit qualities) and principles (implicit qualities) by contrasting preferred and rejected responses.<n>Our results show that rubrics provide scalable alignment signals that narrow the gap between costly human evaluation and automated reward modeling.
  arXiv  Detail & Related papers  (2025-10-09T03:31:26Z)
• RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution [50.171320156632866]
  Reinforcement learning from human feedback offers a promising approach to aligning large language models with human preferences.<n>Current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence.<n>We propose a more fine-grained, token-level guidance approach for RL training.
  arXiv  Detail & Related papers  (2024-11-13T02:45:21Z)
• CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling [27.86204841898399]
  Reward modeling in large language models is susceptible to reward hacking.<n>We propose Context-Aware Reward Modeling (CARMO) to mitigate this problem.<n>We establish a new state-of-the-art performance in zero-shot settings for generative models, achieving a 2.1% improvement on Reward Bench.
  arXiv  Detail & Related papers  (2024-10-28T21:18:49Z)
• RewardBench: Evaluating Reward Models for Language Modeling [100.28366840977966]
  We present RewardBench, a benchmark dataset and code-base for evaluation of reward models. The dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods.
  arXiv  Detail & Related papers  (2024-03-20T17:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.