Fisher Flow Matching for Generative Modeling over Discrete Data

• URL: http://arxiv.org/abs/2405.14664v4
• Date: Wed, 30 Oct 2024 11:01:10 GMT
• Title: Fisher Flow Matching for Generative Modeling over Discrete Data
• Authors: Oscar Davis, Samuel Kessler, Mircea Petrache, İsmail İlkan Ceylan, Michael Bronstein, Avishek Joey Bose,
• Abstract summary: We introduce Fisher-Flow, a novel flow-matching model for discrete data. Fisher-Flow takes a manifestly geometric perspective by considering categorical distributions over discrete data. We prove that the gradient flow induced by Fisher-Flow is optimal in reducing the forward KL divergence.
• Score: 12.69975914345141
• License:
• Abstract: Generative modeling over discrete data has recently seen numerous success stories, with applications spanning language modeling, biological sequence design, and graph-structured molecular data. The predominant generative modeling paradigm for discrete data is still autoregressive, with more recent alternatives based on diffusion or flow-matching falling short of their impressive performance in continuous data settings, such as image or video generation. In this work, we introduce Fisher-Flow, a novel flow-matching model for discrete data. Fisher-Flow takes a manifestly geometric perspective by considering categorical distributions over discrete data as points residing on a statistical manifold equipped with its natural Riemannian metric: the $\textit{Fisher-Rao metric}$. As a result, we demonstrate discrete data itself can be continuously reparameterised to points on the positive orthant of the $d$-hypersphere $\mathbb{S}^d_+$, which allows us to define flows that map any source distribution to target in a principled manner by transporting mass along (closed-form) geodesics of $\mathbb{S}^d_+$. Furthermore, the learned flows in Fisher-Flow can be further bootstrapped by leveraging Riemannian optimal transport leading to improved training dynamics. We prove that the gradient flow induced by Fisher-Flow is optimal in reducing the forward KL divergence. We evaluate Fisher-Flow on an array of synthetic and diverse real-world benchmarks, including designing DNA Promoter, and DNA Enhancer sequences. Empirically, we find that Fisher-Flow improves over prior diffusion and flow-matching models on these benchmarks.

Related papers

• Wasserstein Flow Matching: Generative modeling over families of distributions [13.620905707751747]
  We show how to perform generative modeling over Gaussian distributions, where we generate representations of granular cell states from single-cell genomics data. We also show that WFM can learn flows between high-dimensional and variable sized point-clouds and synthesize cellular microenvironments from spatial transcriptomics datasets.
  arXiv  Detail & Related papers  (2024-11-01T15:55:07Z)
• DeFoG: Discrete Flow Matching for Graph Generation [45.037260759871124]
  We propose DeFoG, a novel framework using discrete flow matching for graph generation. DeFoG employs a flow-based approach that features an efficient linear noising process and a flexible denoising process. We show that DeFoG achieves state-of-the-art results on synthetic and molecular datasets.
  arXiv  Detail & Related papers  (2024-10-05T18:52:54Z)
• Sub-graph Based Diffusion Model for Link Prediction [43.15741675617231]
  Denoising Diffusion Probabilistic Models (DDPMs) represent a contemporary class of generative models with exceptional qualities. We build a novel generative model for link prediction using a dedicated design to decompose the likelihood estimation process via the Bayesian formula. Our proposed method presents numerous advantages: (1) transferability across datasets without retraining, (2) promising generalization on limited training data, and (3) robustness against graph adversarial attacks.
  arXiv  Detail & Related papers  (2024-09-13T02:23:55Z)
• Amortizing intractable inference in diffusion models for vision, language, and control [89.65631572949702]
  This paper studies amortized sampling of the posterior over data, $mathbfxsim prm post(mathbfx)propto p(mathbfx)r(mathbfx)$, in a model that consists of a diffusion generative model prior $p(mathbfx)$ and a black-box constraint or function $r(mathbfx)$. We prove the correctness of a data-free learning objective, relative trajectory balance, for training a diffusion model that samples from
  arXiv  Detail & Related papers  (2024-05-31T16:18:46Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Bayesian Structure Learning with Generative Flow Networks [85.84396514570373]
  In Bayesian structure learning, we are interested in inferring a distribution over the directed acyclic graph (DAG) from data. Recently, a class of probabilistic models, called Generative Flow Networks (GFlowNets), have been introduced as a general framework for generative modeling. We show that our approach, called DAG-GFlowNet, provides an accurate approximation of the posterior over DAGs.
  arXiv  Detail & Related papers  (2022-02-28T15:53:10Z)
• Discrete Denoising Flows [87.44537620217673]
  We introduce a new discrete flow-based model for categorical random variables: Discrete Denoising Flows (DDFs) In contrast with other discrete flow-based models, our model can be locally trained without introducing gradient bias. We show that DDFs outperform Discrete Flows on modeling a toy example, binary MNIST and Cityscapes segmentation maps, measured in log-likelihood.
  arXiv  Detail & Related papers  (2021-07-24T14:47:22Z)
• Flows for simultaneous manifold learning and density estimation [12.451050883955071]
  manifold-learning flows (M-flows) represent datasets with a manifold structure more faithfully. M-flows learn the data manifold and allow for better inference than standard flows in the ambient data space.
  arXiv  Detail & Related papers  (2020-03-31T02:07:48Z)
• Closing the Dequantization Gap: PixelCNN as a Single-Layer Flow [16.41460104376002]
  We introduce subset flows, a class of flows that can transform finite volumes and allow exact computation of likelihoods for discrete data. We identify ordinal discrete autoregressive models, including WaveNets, PixelCNNs and Transformers, as single-layer flows. We demonstrate state-of-the-art results on CIFAR-10 for flow models trained with dequantization.
  arXiv  Detail & Related papers  (2020-02-06T22:58:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.