dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models

• URL: http://arxiv.org/abs/2605.09291v1
• Date: Sun, 10 May 2026 03:36:49 GMT
• Title: dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models
• Abstract summary: We present discrete Flow-GRPO, a unified reinforcement learning framework for discrete flow models.<n>We derive the full trajectory probability for DFMs and formulate denoising as a Markov decision process.<n>We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks.
• Score: 8.198964054238731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of mixture path and a masked source distribution. While several recent works have explored reinforcement learning into dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate denoising as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using FlowGRPO, while also demonstrating strong capabilities on understanding tasks.

Related papers

• Normalizing Flows with Iterative Denoising [32.9706166699763]
  We introduce iterative TARFlow (iTARFlow) to Normalizing Flow generative models.<n>iTARFlow maintains a fully end-to-end, likelihood-based objective during training.<n>It achieves competitive performance across ImageNet resolutions of 64, 128, and 256 pixels.
  arXiv  Detail & Related papers  (2026-04-21T22:52:26Z)
• Efficient Flow Matching using Latent Variables [9.363347684114474]
  We show that $texttLatent-CFM$ exhibits improved generation quality with significantly less training and computation than state-of-the-art flow matching models.<n>We also consider generative modeling of spatial fields stemming from physical processes.
  arXiv  Detail & Related papers  (2025-05-07T14:59:23Z)
• TFG-Flow: Training-free Guidance in Multimodal Generative Flow [73.93071065307782]
  We introduce TFG-Flow, a training-free guidance method for multimodal generative flow.<n>TFG-Flow addresses the curse-of-dimensionality while maintaining the property of unbiased sampling in guiding discrete variables.<n>We show that TFG-Flow has great potential in drug design by generating molecules with desired properties.
  arXiv  Detail & Related papers  (2025-01-24T03:44:16Z)
• Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review [63.31328039424469]
  This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
  arXiv  Detail & Related papers  (2024-07-18T17:35:32Z)
• Fisher Flow Matching for Generative Modeling over Discrete Data [12.69975914345141]
  We introduce Fisher-Flow, a novel flow-matching model for discrete data. Fisher-Flow takes a manifestly geometric perspective by considering categorical distributions over discrete data. We prove that the gradient flow induced by Fisher-Flow is optimal in reducing the forward KL divergence.
  arXiv  Detail & Related papers  (2024-05-23T15:02:11Z)
• Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling [2.1779479916071067]
  We introduce a novel framework that enhances diffusion models by supporting a broader range of forward processes.<n>We also propose a novel parameterization technique for learning the forward process.<n>Results underscore NFDM's versatility and its potential for a wide range of applications.
  arXiv  Detail & Related papers  (2024-04-19T15:10:54Z)
• Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design [37.634098563033795]
  We present a new flow-based model of discrete data that provides the missing link in enabling flow-based generative models. Our key insight is that the discrete equivalent of continuous space flow matching can be realized using Continuous Time Markov Chains. We apply this capability to the task of protein co-design, wherein we learn a model for jointly generating protein structure and sequence.
  arXiv  Detail & Related papers  (2024-02-07T16:15:36Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models [77.83923746319498]
  We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models. We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
  arXiv  Detail & Related papers  (2023-05-29T04:22:57Z)
• Generative Flows with Invertible Attentions [135.23766216657745]
  We introduce two types of invertible attention mechanisms for generative flow models. We exploit split-based attention mechanisms to learn the attention weights and input representations on every two splits of flow feature maps. Our method provides invertible attention modules with tractable Jacobian determinants, enabling seamless integration of it at any positions of the flow-based models.
  arXiv  Detail & Related papers  (2021-06-07T20:43:04Z)
• Normalizing Flows with Multi-Scale Autoregressive Priors [131.895570212956]
  We introduce channel-wise dependencies in their latent space through multi-scale autoregressive priors (mAR) Our mAR prior for models with split coupling flow layers (mAR-SCF) can better capture dependencies in complex multimodal data. We show that mAR-SCF allows for improved image generation quality, with gains in FID and Inception scores compared to state-of-the-art flow-based models.
  arXiv  Detail & Related papers  (2020-04-08T09:07:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.