Proximal Policy Optimization for Amortized Discrete Sampling

• URL: http://arxiv.org/abs/2606.15793v1
• Date: Sun, 14 Jun 2026 12:54:15 GMT
• Title: Proximal Policy Optimization for Amortized Discrete Sampling
• Abstract summary: This paper explores policy gradient algorithms for training policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlowNet) framework.<n>We derive equivalents of standard policy gradient algorithms for training GFlowNets, as well as experimentally explore their various methodological aspects, including baseline training and advantage estimation.<n>Most importantly, our work is the first to derive and successfully apply proximal policy optimization to GFlowNets, showing its improved convergence speed and data efficiency compared to standard GFlowNet training objectives on benchmarks ranging from synthetic energies to molecular graph generation.
• Score: 7.256574710203357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper explores policy gradient algorithms for training stochastic policies to sample from structured discrete probability distributions under the Generative Flow Network (GFlowNet) framework. Building on extensive theoretical connections between GFlowNets and entropy-regularized reinforcement learning, we derive equivalents of standard policy gradient algorithms for training GFlowNets, as well as experimentally explore their various methodological aspects, including baseline training and advantage estimation. Most importantly, our work is the first to derive and successfully apply proximal policy optimization to GFlowNets, showing its improved convergence speed and data efficiency compared to standard GFlowNet training objectives on benchmarks ranging from synthetic energies to molecular graph generation.

Related papers

• ReFPO: Reflow Regularization for Flow Matching Policy Gradients [58.32178725687043]
  We present a simple online RL method that adds explicit Reflow regularization to Flow Matching Policy Gradients.<n>We experimentally demonstrate that ReFPO improves average performance and discretization across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks.
  arXiv  Detail & Related papers  (2026-06-19T04:23:10Z)
• Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization [4.158255103170876]
  GFlowNets are a family of generative models that learn to sample objects proportional to a given reward function.<n>Recent results show a close relationship between GFlowNet training and entropy-regularized reinforcement learning problems.<n>We introduce a simple backward policy optimization algorithm that involves direct sequentially of the value function in an entropy-regularized Markov Decision Process.
  arXiv  Detail & Related papers  (2024-10-20T19:12:14Z)
• GFlowNet Training by Policy Gradients [11.02335801879944]
  We propose a new GFlowNet training framework, with policy-dependent rewards, that bridges keeping flow balance of GFlowNets to optimizing the expected accumulated reward in traditional Reinforcement-Learning (RL)<n>This enables the derivation of new policy-based GFlowNet training methods, in contrast to existing ones resembling value-based RL.
  arXiv  Detail & Related papers  (2024-08-12T01:24:49Z)
• On Generalization for Generative Flow Networks [54.20924253330039]
  Generative Flow Networks (GFlowNets) have emerged as an innovative learning paradigm designed to address the challenge of sampling from an unnormalized probability distribution. This paper attempts to formalize generalization in the context of GFlowNets, to link generalization with stability, and also to design experiments that assess the capacity of these models to uncover unseen parts of the reward function.
  arXiv  Detail & Related papers  (2024-07-03T13:42:21Z)
• Random Policy Evaluation Uncovers Policies of Generative Flow Networks [12.294107455811496]
  Generative Flow Network (GFlowNet) is a probabilistic framework in which an agent learns a policy and flow functions.<n>In this paper, we reveal a fundamental connection between GFlowNets and one RL's most basic components -- policy evaluation.<n>We introduce a rectified random policy evaluation algorithm, which achieves the same reward-matching effect as GFlowNets.
  arXiv  Detail & Related papers  (2024-06-04T11:11:53Z)
• Generative Flow Networks as Entropy-Regularized RL [4.857649518812728]
  generative flow networks (GFlowNets) are a method of training a policy to sample compositional objects with proportional probabilities to a given reward via a sequence of actions. We demonstrate how the task of learning a generative flow network can be efficiently as an entropy-regularized reinforcement learning problem. Contrary to previously reported results, we show that entropic RL approaches can be competitive against established GFlowNet training methods.
  arXiv  Detail & Related papers  (2023-10-19T17:31:40Z)
• Pre-Training and Fine-Tuning Generative Flow Networks [61.90529626590415]
  We introduce a novel approach for reward-free pre-training of GFlowNets. By framing the training as a self-supervised problem, we propose an outcome-conditioned GFlowNet that learns to explore the candidate space. We show that the pre-trained OC-GFN model can allow for a direct extraction of a policy capable of sampling from any new reward functions in downstream tasks.
  arXiv  Detail & Related papers  (2023-10-05T09:53:22Z)
• Thompson sampling for improved exploration in GFlowNets [75.89693358516944]
  Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over compositional objects as a sequential decision-making problem with a learnable action policy. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.
  arXiv  Detail & Related papers  (2023-06-30T14:19:44Z)
• Stochastic Generative Flow Networks [89.34644133901647]
  Generative Flow Networks (or GFlowNets) learn to sample complex structures through the lens of "inference as control" Existing GFlowNets can be applied only to deterministic environments, and fail in more general tasks with dynamics. This paper introduces GFlowNets, a new algorithm that extends GFlowNets to environments.
  arXiv  Detail & Related papers  (2023-02-19T03:19:40Z)
• Distributional GFlowNets with Quantile Flows [73.73721901056662]
  Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns a policy for generating complex structure through a series of decision-making steps. In this work, we adopt a distributional paradigm for GFlowNets, turning each flow function into a distribution, thus providing more informative learning signals during training. Our proposed textitquantile matching GFlowNet learning algorithm is able to learn a risk-sensitive policy, an essential component for handling scenarios with risk uncertainty.
  arXiv  Detail & Related papers  (2023-02-11T22:06:17Z)
• Improving Generative Flow Networks with Path Regularization [8.848799220256366]
  Generative Flow Networks (GFlowNets) are recently proposed models for learning policies that generate compositional objects by sequences of actions with the probability proportional to a given reward function. In this work, we propose a novel path regularization method based on optimal transport theory that places prior constraints on the underlying structure of the GFlowNets.
  arXiv  Detail & Related papers  (2022-09-29T20:54:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.