NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning
- URL: http://arxiv.org/abs/2606.27771v1
- Date: Fri, 26 Jun 2026 06:56:54 GMT
- Title: NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning
- Abstract summary: Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality.<n>We identify a simple structural signature of this drift across three post-training methods (NFT, AWM, DPO)<n>We find that RL fine-tuning inflates the per-step velocity norm $|v_|$ by $5%$ to $15%$ relative to the reference.
- Score: 42.29085249260694
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Reinforcement learning (RL) post-training improves the reward alignment of flow-based generators, but often degrades perceptual quality in ways that are not captured by the reward proxy. We identify a simple structural signature of this drift: across three post-training methods (NFT, AWM, DPO), RL fine-tuning inflates the per-step velocity norm $\|v_θ\|$ by $5\%$ to $15\%$ relative to the reference. A form of norm inflation has been studied in classifier-free guidance (CFG), where rescaling the velocity back to a reference norm at inference time can mitigate the resulting artifacts. However, this inference-time correction does not transfer cleanly to RL: rescaling $v_θ$ to match $\|v_{\text{ref}}\|$ at inference time neither improves reward nor fixes the quality degradation, because the inflation is co-adapted into the model weights. Furthermore, an adjoint sensitivity analysis shows that velocity magnitude rescaling carries no coherent first-order reward signal at the batch level, indicating that suppressing norm inflation is unlikely to remove a consistently reward-carrying component. Since inference-time renormalization fails while norm suppression carries no reward cost, training-time intervention is the appropriate strategy. Together, these findings motivate \methodname, a hinge penalty that activates only when $\|v_θ\|$ exceeds $\|v_{\text{ref}}\|$ and composes additively with any velocity-local base loss. Across two base models, three post-training methods, and two reward proxies, \methodname consistently improves MLLM-judged image quality and forensic realism while preserving reward, with gains that amplify under few-step inference and are not explained by early stopping.
Related papers
- Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework [6.490241400619907]
Reinforcement learning from internal feedback has recently emerged as a scalable unsupervised alternative.<n>We propose a multi-reward RLIF framework that decomposes the training signal into two complementary components.<n>We show that our method can support stable long-horizon reasoning without relying on external ground-truth supervision.
arXiv Detail & Related papers (2026-05-21T15:30:47Z) - When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR [46.88928401641895]
Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant paradigm for advanced reasoning in Large Language Models (LLMs)<n>In RLVR, reuse each rollout batch for multiple gradient updates amplifies policy shift, leading to severe performance degradation.<n>We identify the textitDisproportionate Weight Divergence (DWD) phenomenon: performance degradation is synchronized a sharp surge in the textttlm_head weight change, while intermediate layers remain stable.<n>We propose textitDynamic Gradient Gating (DGG), a lightweight intervention
arXiv Detail & Related papers (2026-05-19T06:23:43Z) - Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
arXiv Detail & Related papers (2026-05-11T03:15:37Z) - MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
arXiv Detail & Related papers (2026-05-07T16:20:42Z) - Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
arXiv Detail & Related papers (2026-04-14T18:19:54Z) - Improved Mean Flows: On the Challenges of Fastforward Generative Models [81.10827083963655]
MeanFlow (MF) has recently been established as a framework for one-step generative modeling.<n>Here, we address key challenges in both the training objective and the guidance mechanism.<n>Our reformulation yields a more standard regression problem and improves the training stability.<n>Overall, our $textbfimproved MeanFlow$ ($textbfiMF$) method, trained entirely from scratch, achieves $textbf1.72$ FID with a single function evaluation (1-NFE) on ImageNet 256$times$256.
arXiv Detail & Related papers (2025-12-01T18:59:49Z) - Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
arXiv Detail & Related papers (2025-10-01T13:56:44Z) - Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless $l^p$ Norm Solution for Fast Adversarial Training [6.393645655578601]
We propose a novel solution that controls the $lp$ training norm to mitigate Catastrophic Overfitting (CO)<n>Our study is motivated by the empirical observation that CO is more prevalent under the $linfty$ norm than the $l2$ norm.<n>By quantifying gradient concentration through Participation Ratio and entropy measures, we develop an adaptive $lp$-FGSM that automatically tunes the training norm based on gradient information.
arXiv Detail & Related papers (2025-05-05T04:41:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.