Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces

• URL: http://arxiv.org/abs/2412.01019v1
• Date: Mon, 02 Dec 2024 00:35:29 GMT
• Title: Energy-Based Modelling for Discrete and Mixed Data via Heat Equations on Structured Spaces
• Authors: Tobias Schröder, Zijing Ou, Yingzhen Li, Andrew B. Duncan,
• Abstract summary: Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. We propose to train discrete EBMs with Energy Discrepancy, a loss function which only requires the evaluation of the energy function at data points.
• Score: 19.92604781654767
• License:
• Abstract: Energy-based models (EBMs) offer a flexible framework for probabilistic modelling across various data domains. However, training EBMs on data in discrete or mixed state spaces poses significant challenges due to the lack of robust and fast sampling methods. In this work, we propose to train discrete EBMs with Energy Discrepancy, a loss function which only requires the evaluation of the energy function at data points and their perturbed counterparts, thus eliminating the need for Markov chain Monte Carlo. We introduce perturbations of the data distribution by simulating a diffusion process on the discrete state space endowed with a graph structure. This allows us to inform the choice of perturbation from the structure of the modelled discrete variable, while the continuous time parameter enables fine-grained control of the perturbation. Empirically, we demonstrate the efficacy of the proposed approaches in a wide range of applications, including the estimation of discrete densities with non-binary vocabulary and binary image modelling. Finally, we train EBMs on tabular data sets with applications in synthetic data generation and calibrated classification.

Related papers

• Discrete vs. Continuous Trade-offs for Generative Models [0.0]
  This work explores the theoretical and practical foundations of denoising diffusion probabilistic models (DDPMs) DDPMs and score-based generative models, which leverage processes and Brownian motion to model complex data distributions.
  arXiv  Detail & Related papers  (2024-12-26T08:14:27Z)
• A Data-Driven Framework for Discovering Fractional Differential Equations in Complex Systems [8.206685537936078]
  This study introduces a stepwise data-driven framework for discovering fractional differential equations (FDEs) directly from data. Our framework applies deep neural networks as surrogate models for denoising and reconstructing sparse and noisy observations. We validate the framework across various datasets, including synthetic anomalous diffusion data and experimental data on the creep behavior of frozen soils.
  arXiv  Detail & Related papers  (2024-12-05T08:38:30Z)
• Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations [1.371691382573869]
  We develop and enhance score-based diffusion models in field reconstruction tasks. We introduce a condition encoding approach to construct a tractable mapping mapping between observed and unobserved regions. We demonstrate the ability of the model to capture possible reconstructions and improve the accuracy of fused results.
  arXiv  Detail & Related papers  (2024-08-30T19:46:23Z)
• Constrained Diffusion Models via Dual Training [80.03953599062365]
  Diffusion processes are prone to generating samples that reflect biases in a training dataset. We develop constrained diffusion models by imposing diffusion constraints based on desired distributions. We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
  arXiv  Detail & Related papers  (2024-08-27T14:25:42Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Synthetic location trajectory generation using categorical diffusion models [50.809683239937584]
  Diffusion models (DPMs) have rapidly evolved to be one of the predominant generative models for the simulation of synthetic data. We propose using DPMs for the generation of synthetic individual location trajectories (ILTs) which are sequences of variables representing physical locations visited by individuals.
  arXiv  Detail & Related papers  (2024-02-19T15:57:39Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Training Discrete Energy-Based Models with Energy Discrepancy [18.161764926125066]
  Training energy-based models (EBMs) on discrete spaces is challenging because sampling over such spaces can be difficult. We propose to train discrete EBMs with energy discrepancy (ED), a novel type of contrastive loss functional. We demonstrate their relative performance on lattice Ising models, binary synthetic data, and discrete image data sets.
  arXiv  Detail & Related papers  (2023-07-14T19:38:05Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• $\mathcal{F}$-EBM: Energy Based Learning of Functional Data [1.0896567381206714]
  Energy-Based Models (EBMs) have proven to be a highly effective approach for modelling densities on finite-dimensional spaces. We present a novel class of EBM which is able to learn distributions of functions from functional samples evaluated at finitely many points.
  arXiv  Detail & Related papers  (2022-02-04T01:01:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.