Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training

• URL: http://arxiv.org/abs/2606.21498v1
• Date: Fri, 19 Jun 2026 14:50:22 GMT
• Title: Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training
• Abstract summary: Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging.<n>We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence.<n>Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways.
• Score: 8.16921896191724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging. GRPO-style online reinforcement learning provides an effective framework; however, existing methods typically treat reference-policy divergence as fixed, despite its direct impact on policy optimization. We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence, for GRPO-style autoregressive T2I alignment. Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways. In particular, under the sampled-token shaping form used, JS regularization achieves a favorable trade-off by mitigating uniform bias relative to the reference policy while still discouraging large deviations. Extensive experiments on LlamaGen and Janus-7B show that JS divergence achieves the strongest or highly competitive optimization performance on most evaluation metrics while maintaining favorable generation diversity. The code is available at https://github.com/tuoyou-hao/BPD-GRPO.

Related papers

• UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation [51.41441081823758]
  Unified models capable of interleaved generation have emerged as a promising paradigm.<n>We propose a unified reinforcement learning framework tailored for interleaved generation.<n>Our experiments demonstrate that this unified training recipe significantly enhances image generation quality through reasoning.
  arXiv  Detail & Related papers  (2026-03-24T17:59:17Z)
• Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards [16.135177543347773]
  Autoregressive (AR) models are highly effective for image generation, yet their standard maximum-likelihood estimation training lacks direct optimization for sample quality and diversity.<n>We propose a lightweight RL framework that casts token-based AR inference as a Markov Decision Process, optimized via Group Relative Policy Optimization.<n>Our core contribution is the introduction of a novel distribution-level Leave-One-Out FID (LOO-FID) reward.
  arXiv  Detail & Related papers  (2026-03-24T11:28:36Z)
• DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay [57.80564154223355]
  Existing experience replay methods address this by reusing accurate samples for direct policy updates.<n>We argue that historical data should prioritize sustaining diversity rather than simply reinforcing accuracy.<n>We propose Dynamic Jensen-Shannon Replay (LEPJR), a simple yet effective regularization framework.
  arXiv  Detail & Related papers  (2026-03-17T06:20:56Z)
• When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO [18.988527161000203]
  Group Relative Policy Optimization (GRPO) has emerged as an effective method for training reasoning models.<n>We present a contrastive reformulation of GRPO, showing that the GRPO objective implicitly maximizes the margin between the policy ratios of correct and incorrect samples.<n>We propose Bilateral Context Conditioning (BICC), a mechanism that allows the model to cross-reference successful and failed reasoning traces.
  arXiv  Detail & Related papers  (2026-03-13T16:25:02Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Anchoring Values in Temporal and Group Dimensions for Flow Matching Model Alignment [61.80228667422234]
  VGPO redefines value estimation across both temporal and group dimensions.<n>It transforms the sparse terminal reward into dense, process-aware value estimates.<n>It replaces standard group normalization with a novel process enhanced by absolute values to maintain a stable optimization signal.
  arXiv  Detail & Related papers  (2025-12-13T16:31:26Z)
• Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation [18.178060190776858]
  We introduce BPGO, a novel extension of GRPO that explicitly models reward uncertainty through a semantic prior anchor.<n>BPGO delivers consistently stronger semantic alignment, enhanced perceptual fidelity, and faster convergence than standard GRPO and recent variants.
  arXiv  Detail & Related papers  (2025-11-24T09:29:30Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)
• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO [22.00487909203855]
  Group Relative Policy Optimization fails to update a policy when all responses within a group are incorrect.<n>This limitation underscores a key gap between artificial and human intelligence.<n>We introduce a simple framework that mitigates the all-negative-sample issue by incorporating response diversity within groups.
  arXiv  Detail & Related papers  (2025-05-16T18:02:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.