Loss-Guided Auxiliary Agents for Overcoming Mode Collapse in GFlowNets

• URL: http://arxiv.org/abs/2505.15251v2
• Date: Mon, 10 Nov 2025 06:28:26 GMT
• Title: Loss-Guided Auxiliary Agents for Overcoming Mode Collapse in GFlowNets
• Authors: Idriss Malek, Aya Laajil, Abhijith Sharma, Eric Moulines, Salem Lahlou,
• Abstract summary: Loss-Guided GFlowNets (LGGFN) is a novel approach where an auxiliary GFlowNet's exploration is textbfdirectly driven by the main GFlowNet's training loss<n>This targeted exploration significantly accelerates the discovery of diverse, high-reward samples.
• Score: 22.653875450786444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although Generative Flow Networks (GFlowNets) are designed to capture multiple modes of a reward function, they often suffer from mode collapse in practice, getting trapped in early-discovered modes and requiring prolonged training to find diverse solutions. Existing exploration techniques often rely on heuristic novelty signals. We propose Loss-Guided GFlowNets (LGGFN), a novel approach where an auxiliary GFlowNet's exploration is \textbf{directly driven by the main GFlowNet's training loss}. By prioritizing trajectories where the main model exhibits \textbf{high loss}, LGGFN focuses sampling on poorly understood regions of the state space. This targeted exploration significantly accelerates the discovery of diverse, high-reward samples. Empirically, across \textbf{diverse benchmarks} including grid environments, structured sequence generation, Bayesian structure learning, and biological sequence design, LGGFN consistently \textbf{outperforms} baselines in exploration efficiency and sample diversity. For instance, on a challenging sequence generation task, it discovered over 40 times more unique valid modes while simultaneously reducing the exploration error metric by approximately 99\%.

Related papers

• Boosted GFlowNets: Improving Exploration via Sequential Learning [13.119757506183392]
  Boosted GFlowNets are a method that sequentially trains an ensemble of GFlowNets, each optimizing a residual reward that compensates for the mass already captured by previous models.<n>We show that Boosted GFlowNets achieve substantially better exploration and sample diversity on multimodal synthetic benchmarks and peptide design tasks.
  arXiv  Detail & Related papers  (2025-11-12T19:30:11Z)
• MG2FlowNet: Accelerating High-Reward Sample Generation via Enhanced MCTS and Greediness Control [19.49552596070782]
  Generative Flow Networks (GFlowNets) have emerged as a powerful tool for generating diverse and high-reward structured objects by learning to sample from a distribution proportional to a given reward function.<n>In this work, we integrate an enhanced Monte Carlo Tree Search (MCTS) into the GFlowNets sampling process to balance exploration and exploitation adaptively.<n>Our method can not only accelerate the speed of discovering high-reward regions but also continuously generate high-reward samples, while preserving the diversity of the generative distribution.
  arXiv  Detail & Related papers  (2025-10-01T12:09:04Z)
• MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets [1.892757060653176]
  We introduce Adapted Metadynamics, a variant of metadynamics that can be applied to arbitrary black-box reward functions on continuous domains.<n>We show several continuous domains where the resulting algorithm, MetaGFN, accelerates convergence to the target distribution and discovers more distant reward modes than previous off-policy exploration strategies used for GFlowNets.
  arXiv  Detail & Related papers  (2024-08-28T16:19:35Z)
• Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets [27.33222647437964]
  Generative Flow Networks (GFlowNets) have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates.<n>However, training such models is challenging due to extremely sparse rewards.<n>We propose a novel method called textbfRetrospective textbfBackward textbfSynthesis (textbfRBS) to address these problems.
  arXiv  Detail & Related papers  (2024-06-03T09:44:10Z)
• Pre-Training and Fine-Tuning Generative Flow Networks [61.90529626590415]
  We introduce a novel approach for reward-free pre-training of GFlowNets. By framing the training as a self-supervised problem, we propose an outcome-conditioned GFlowNet that learns to explore the candidate space. We show that the pre-trained OC-GFN model can allow for a direct extraction of a policy capable of sampling from any new reward functions in downstream tasks.
  arXiv  Detail & Related papers  (2023-10-05T09:53:22Z)
• Local Search GFlowNets [85.0053493167887]
  Generative Flow Networks (GFlowNets) are amortized sampling methods that learn a distribution over discrete objects proportional to their rewards. GFlowNets exhibit a remarkable ability to generate diverse samples, yet occasionally struggle to consistently produce samples with high rewards due to over-exploration on wide sample space. This paper proposes to train GFlowNets with local search, which focuses on exploiting high-rewarded sample space to resolve this issue.
  arXiv  Detail & Related papers  (2023-10-04T10:27:17Z)
• Thompson sampling for improved exploration in GFlowNets [75.89693358516944]
  Generative flow networks (GFlowNets) are amortized variational inference algorithms that treat sampling from a distribution over compositional objects as a sequential decision-making problem with a learnable action policy. We show in two domains that TS-GFN yields improved exploration and thus faster convergence to the target distribution than the off-policy exploration strategies used in past work.
  arXiv  Detail & Related papers  (2023-06-30T14:19:44Z)
• Stochastic Generative Flow Networks [89.34644133901647]
  Generative Flow Networks (or GFlowNets) learn to sample complex structures through the lens of "inference as control" Existing GFlowNets can be applied only to deterministic environments, and fail in more general tasks with dynamics. This paper introduces GFlowNets, a new algorithm that extends GFlowNets to environments.
  arXiv  Detail & Related papers  (2023-02-19T03:19:40Z)
• A theory of continuous generative flow networks [104.93913776866195]
  Generative flow networks (GFlowNets) are amortized variational inference algorithms that are trained to sample from unnormalized target distributions. We present a theory for generalized GFlowNets, which encompasses both existing discrete GFlowNets and ones with continuous or hybrid state spaces.
  arXiv  Detail & Related papers  (2023-01-30T00:37:56Z)
• GFlowCausal: Generative Flow Networks for Causal Discovery [27.51595081346858]
  We propose a novel approach to learning a Directed Acyclic Graph (DAG) from observational data called GFlowCausal. GFlowCausal aims to learn the best policy to generate high-reward DAGs by sequential actions with probabilities proportional to predefined rewards. We conduct extensive experiments on both synthetic and real datasets, and results show the proposed approach to be superior and also performs well in a large-scale setting.
  arXiv  Detail & Related papers  (2022-10-15T04:07:39Z)
• Learning GFlowNets from partial episodes for improved convergence and stability [56.99229746004125]
  Generative flow networks (GFlowNets) are algorithms for training a sequential sampler of discrete objects under an unnormalized target density. Existing training objectives for GFlowNets are either local to states or transitions, or propagate a reward signal over an entire sampling trajectory. Inspired by the TD($lambda$) algorithm in reinforcement learning, we introduce subtrajectory balance or SubTB($lambda$), a GFlowNet training objective that can learn from partial action subsequences of varying lengths.
  arXiv  Detail & Related papers  (2022-09-26T15:44:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.