The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL

• URL: http://arxiv.org/abs/2606.19162v1
• Date: Wed, 17 Jun 2026 15:01:33 GMT
• Title: The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL
• Abstract summary: Discriminator-Guided RL (DRL) trains a discriminator to separate data from base-model samples in a pretrained representation space.<n>It reduces guidance-free FID (e.g., $9.38 to 2.62$ on SiT) and semantic-space FD (e.g., $88.2 to 19.3$ on DINOv3 for SiT)<n>It also yields a better frontier between preference reward and image fidelity under subsequent preference-based post-training.
• Score: 19.556118547263928
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself. We argue that this reflects a structural mismatch. Matching losses measure $\ell_2$ regression error on the velocity or score field under training-time marginals, a proxy poorly aligned with the visual and semantic properties that determine sample quality at inference. Given a reward aligned with these properties, RL sidesteps the mismatch by evaluating the model on its own samples and following the reward landscape directly. The challenge is to obtain such a reward without relying on human preferences, which are expensive and conflate data realism with annotator inclinations. We propose Discriminator-Guided RL (DRL). DRL trains a discriminator to separate data from base-model samples in a pretrained representation space and uses its logit as the reward in KL-regularized RL. The pretrained space restricts the discriminator to perceptually meaningful directions, and the logit estimates the log-likelihood ratio between data and model, which is the optimal reward for targeting the data distribution. Across SiT, JiT, REPA, and RAE, DRL reduces guidance-free FID (e.g., $9.38 \to 2.62$ on SiT) and semantic-space FD (e.g., $88.2 \to 19.3$ on DINOv3 for SiT), with consistent gains across all backbones, and improves human-preference rewards without training on them. It also yields a better Pareto frontier between preference reward and image fidelity under subsequent preference-based post-training, increasing alignment while reducing low-level artifacts such as oversaturation and excessive brightness.

Related papers

• Optimizing Visual Generative Models via Distribution-wise Rewards [57.46109819595665]
  We present a novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions.<n>Our approach significantly improves FID-50K across various base models, from 5.77 for SiT and from 3.74 to 3.52 for EDM2.
  arXiv  Detail & Related papers  (2026-07-02T15:08:56Z)
• Variance-aware Reward Modeling with Anchor Guidance [10.561814492691534]
  We propose Anchor-guided Variance-aware Reward Modeling, a framework that resolves non-identifiability.<n>Across simulation studies and four real-world diverging-preference datasets, our method consistently improves reward modeling performance.
  arXiv  Detail & Related papers  (2026-05-12T09:46:26Z)
• Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models [44.082049677105964]
  Diffusion and flow-matching models scale because pretraining is supervised regression.<n>We show that the structure extends to RL post-training.<n>Under KL-regularized reward, the optimal generative process tilts the clean-endpoint distribution towards samples with higher reward.<n>We derive Reinforce Adjoint Matching (RAM): a consistency loss that corrects the pretraining target with the reward.
  arXiv  Detail & Related papers  (2026-05-11T15:56:29Z)
• ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment [49.93516092785756]
  Current reward modeling is heavily contingent upon experimental feedback data with high collection costs.<n>ImplicitRM aims to learn unbiased reward models from implicit preference data.<n> Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets.
  arXiv  Detail & Related papers  (2026-03-24T13:32:14Z)
• Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models [50.99097734404912]
  We show that RL-friendly models exhibit intra-class compactness and inter-class separation in their probability assignments to correct vs. incorrect responses.<n>Experiments across six mathematical benchmarks show consistent improvements across all model families, with gains up to 5.9 points on AIME24.
  arXiv  Detail & Related papers  (2026-01-11T13:34:44Z)
• Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values [53.72318444646282]
  We propose Reinforcement Learning with Explicit Human Values (RLEV)<n>RLEV aligns Large Language Model (LLM) optimization directly with quantifiable human value signals.<n>We show RLEV consistently outperforms correctness-only baselines across multiple RL algorithms and model scales.
  arXiv  Detail & Related papers  (2025-10-23T04:15:22Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback [12.158181906895186]
  Reinforcement learning with human feedback has emerged as a central paradigm for aligning large language models with human preferences.<n>We investigate exploration principles for online RLHF, where one seeks to refine both the reward model and the policy in a data-efficient manner.<n>Motivated by this insight, we propose a new exploration scheme that directs preference queries toward reducing uncertainty in reward differences.
  arXiv  Detail & Related papers  (2025-09-26T17:57:17Z)
• SPARK: Synergistic Policy And Reward Co-Evolving Framework [84.22494672256894]
  We introduce the Synergistic Policy And Reward Co-Evolving Framework (SPARK), an efficient, on-policy, and stable method that builds on RLVR.<n>Instead of discarding rollouts and correctness data, SPARK recycles this valuable information to simultaneously train the model itself as a generative reward model.<n>We show that SPARK achieves significant performance gains on multiple LLM and LVLM models and multiple reasoning, reward models, and general benchmarks.
  arXiv  Detail & Related papers  (2025-09-26T17:50:12Z)
• RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards [29.53129965767002]
  We propose Reinforcement Learning with Binary Flexible Feedback (RLBFF)<n>RLBFF combines the versatility of human-driven preferences with the precision of rule-based verification.<n>We show that Reward Models trained in this manner can outperform Bradley-Terry models when matched for data.
  arXiv  Detail & Related papers  (2025-09-25T16:19:06Z)
• Best Policy Learning from Trajectory Preference Feedback [11.896067099790962]
  Preference-based Reinforcement Learning (PbRL) offers a more robust alternative.<n>We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models.<n>We propose Posterior Sampling for Preference Learning ($mathsfPSPL$), a novel algorithm inspired by Top-Two Thompson Sampling.
  arXiv  Detail & Related papers  (2025-01-31T03:55:10Z)
• In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning [15.369324784520538]
  We propose In-Dataset Trajectory Return Regularization (DTR) for offline preference-based reinforcement learning.<n>DTR mitigates the risk of learning inaccurate trajectory stitching under reward bias.<n>We also introduce an ensemble normalization technique that effectively integrates multiple reward models.
  arXiv  Detail & Related papers  (2024-12-12T09:35:47Z)
• Robust Reinforcement Learning from Corrupted Human Feedback [86.17030012828003]
  Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. We propose a robust RLHF approach -- $R3M$, which models the potentially corrupted preference label as sparse outliers. Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R3M$ improves robustness of the reward against several types of perturbations to the preference data.
  arXiv  Detail & Related papers  (2024-06-21T18:06:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.