Evolution Guided Generative Flow Networks

• URL: http://arxiv.org/abs/2402.02186v1
• Date: Sat, 3 Feb 2024 15:28:53 GMT
• Title: Evolution Guided Generative Flow Networks
• Authors: Zarif Ikram, Ling Pan, Dianbo Liu
• Abstract summary: Generative Flow Networks (GFlowNets) learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. We propose Evolution guided generative flow networks (EGFN), a simple but powerful augmentation to the GFlowNets training using Evolutionary algorithms (EA)
• Score: 11.609895436955242
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative Flow Networks (GFlowNets) are a family of probabilistic generative models that learn to sample compositional objects proportional to their rewards. One big challenge of GFlowNets is training them effectively when dealing with long time horizons and sparse rewards. To address this, we propose Evolution guided generative flow networks (EGFN), a simple but powerful augmentation to the GFlowNets training using Evolutionary algorithms (EA). Our method can work on top of any GFlowNets training objective, by training a set of agent parameters using EA, storing the resulting trajectories in the prioritized replay buffer, and training the GFlowNets agent using the stored trajectories. We present a thorough investigation over a wide range of toy and real-world benchmark tasks showing the effectiveness of our method in handling long trajectories and sparse rewards.

Related papers

• Improving GFlowNets with Monte Carlo Tree Search [6.497027864860203]
  Recent studies have revealed strong connections between GFlowNets and entropy-regularized reinforcement learning. We propose to enhance planning capabilities of GFlowNets by applying Monte Carlo Tree Search (MCTS) Our experiments demonstrate that this approach improves the sample efficiency of GFlowNet training and the generation fidelity of pre-trained GFlowNet models.
  arXiv  Detail & Related papers  (2024-06-19T15:58:35Z)
• Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets [27.33222647437964]
  Generative Flow Networks (GFlowNets) are amortized sampling methods for learning a policy to sequentially generate objects with probabilities to their rewards. GFlowNets exhibit a remarkable ability to generate diverse sets of high-reward proportional objects, in contrast to standard reinforcement learning approaches. Recent works have arisen for learning goal-conditioned GFlowNets to acquire various useful properties, aiming to train a single GFlowNet capable of achieving different goals as the task specifies. We propose a novel method named Retrospective Backward Synthesis (RBS) to address these challenges. Specifically, RBS synthesizes a new backward trajectory
  arXiv  Detail & Related papers  (2024-06-03T09:44:10Z)
• Generative Flow Networks as Entropy-Regularized RL [4.857649518812728]
  generative flow networks (GFlowNets) are a method of training a policy to sample compositional objects with proportional probabilities to a given reward via a sequence of actions. We demonstrate how the task of learning a generative flow network can be efficiently as an entropy-regularized reinforcement learning problem. Contrary to previously reported results, we show that entropic RL approaches can be competitive against established GFlowNet training methods.
  arXiv  Detail & Related papers  (2023-10-19T17:31:40Z)
• Pre-Training and Fine-Tuning Generative Flow Networks [61.90529626590415]
  We introduce a novel approach for reward-free pre-training of GFlowNets. By framing the training as a self-supervised problem, we propose an outcome-conditioned GFlowNet that learns to explore the candidate space. We show that the pre-trained OC-GFN model can allow for a direct extraction of a policy capable of sampling from any new reward functions in downstream tasks.
  arXiv  Detail & Related papers  (2023-10-05T09:53:22Z)
• Expected flow networks in stochastic environments and two-player zero-sum games [63.98522423072093]
  Generative flow networks (GFlowNets) are sequential sampling models trained to match a given distribution. We propose expected flow networks (EFlowNets) which extend GFlowNets to environments. We show that EFlowNets outperform other GFlowNet formulations in tasks such as protein design. We then extend the concept of EFlowNets to adversarial environments, proposing adversarial flow networks (AFlowNets) for two-player zero-sum games.
  arXiv  Detail & Related papers  (2023-10-04T12:50:29Z)
• Towards Understanding and Improving GFlowNet Training [71.85707593318297]
  We introduce an efficient evaluation strategy to compare the learned sampling distribution to the target reward distribution. We propose prioritized replay training of high-reward $x$, relative edge flow policy parametrization, and a novel guided trajectory balance objective.
  arXiv  Detail & Related papers  (2023-05-11T22:50:41Z)
• Distributional GFlowNets with Quantile Flows [73.73721901056662]
  Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns a policy for generating complex structure through a series of decision-making steps. In this work, we adopt a distributional paradigm for GFlowNets, turning each flow function into a distribution, thus providing more informative learning signals during training. Our proposed textitquantile matching GFlowNet learning algorithm is able to learn a risk-sensitive policy, an essential component for handling scenarios with risk uncertainty.
  arXiv  Detail & Related papers  (2023-02-11T22:06:17Z)
• Generative Augmented Flow Networks [88.50647244459009]
  We propose Generative Augmented Flow Networks (GAFlowNets) to incorporate intermediate rewards into GFlowNets. GAFlowNets can leverage edge-based and state-based intrinsic rewards in a joint way to improve exploration.
  arXiv  Detail & Related papers  (2022-10-07T03:33:56Z)
• Learning GFlowNets from partial episodes for improved convergence and stability [56.99229746004125]
  Generative flow networks (GFlowNets) are algorithms for training a sequential sampler of discrete objects under an unnormalized target density. Existing training objectives for GFlowNets are either local to states or transitions, or propagate a reward signal over an entire sampling trajectory. Inspired by the TD($lambda$) algorithm in reinforcement learning, we introduce subtrajectory balance or SubTB($lambda$), a GFlowNet training objective that can learn from partial action subsequences of varying lengths.
  arXiv  Detail & Related papers  (2022-09-26T15:44:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.