Concrete Score Matching: Generalized Score Matching for Discrete Data
- URL: http://arxiv.org/abs/2211.00802v1
- Date: Wed, 2 Nov 2022 00:41:37 GMT
- Title: Concrete Score Matching: Generalized Score Matching for Discrete Data
- Authors: Chenlin Meng, Kristy Choi, Jiaming Song, Stefano Ermon
- Abstract summary: "Concrete score" is a generalization of the (Stein) score for discrete settings.
"Concrete Score Matching" is a framework to learn such scores from samples.
- Score: 109.12439278055213
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Representing probability distributions by the gradient of their density
functions has proven effective in modeling a wide range of continuous data
modalities. However, this representation is not applicable in discrete domains
where the gradient is undefined. To this end, we propose an analogous score
function called the "Concrete score", a generalization of the (Stein) score for
discrete settings. Given a predefined neighborhood structure, the Concrete
score of any input is defined by the rate of change of the probabilities with
respect to local directional changes of the input. This formulation allows us
to recover the (Stein) score in continuous domains when measuring such changes
by the Euclidean distance, while using the Manhattan distance leads to our
novel score function in discrete domains. Finally, we introduce a new framework
to learn such scores from samples called Concrete Score Matching (CSM), and
propose an efficient training objective to scale our approach to high
dimensions. Empirically, we demonstrate the efficacy of CSM on density
estimation tasks on a mixture of synthetic, tabular, and high-dimensional image
datasets, and demonstrate that it performs favorably relative to existing
baselines for modeling discrete data.
Related papers
- Spatially-Aware Diffusion Models with Cross-Attention for Global Field Reconstruction with Sparse Observations [1.371691382573869]
We develop and enhance score-based diffusion models in field reconstruction tasks.
We introduce a condition encoding approach to construct a tractable mapping mapping between observed and unobserved regions.
We demonstrate the ability of the model to capture possible reconstructions and improve the accuracy of fused results.
arXiv Detail & Related papers (2024-08-30T19:46:23Z) - Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data.
Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
arXiv Detail & Related papers (2024-04-24T09:04:36Z) - Stochastic interpolants with data-dependent couplings [31.854717378556334]
We use the framework of interpolants to formalize how to itcouple the base and the target densities.
We show that these transport maps can be learned by solving a simple square loss regression problem analogous to the standard independent setting.
arXiv Detail & Related papers (2023-10-05T17:46:31Z) - Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain.
We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions.
We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
arXiv Detail & Related papers (2022-11-30T05:33:29Z) - Nonparametric Probabilistic Regression with Coarse Learners [1.8275108630751844]
We show that we can compute precise conditional densities with minimal assumptions on the shape or form of the density.
We demonstrate this approach on a variety of datasets and show competitive performance, particularly on larger datasets.
arXiv Detail & Related papers (2022-10-28T16:25:26Z) - FP-Diffusion: Improving Score-based Diffusion Models by Enforcing the
Underlying Score Fokker-Planck Equation [72.19198763459448]
We learn a family of noise-conditional score functions corresponding to the data density perturbed with increasingly large amounts of noise.
These perturbed data densities are linked together by the Fokker-Planck equation (FPE), a partial differential equation (PDE) governing the spatial-temporal evolution of a density.
We derive a corresponding equation called the score FPE that characterizes the noise-conditional scores of the perturbed data densities.
arXiv Detail & Related papers (2022-10-09T16:27:25Z) - Denoising Likelihood Score Matching for Conditional Score-based Data
Generation [22.751924447125955]
We propose a novel training objective called Denoising Likelihood Score Matching (DLSM) loss to match the gradients of the true log likelihood density.
Our experimental evidence shows that the proposed method outperforms the previous methods noticeably in terms of several key evaluation metrics.
arXiv Detail & Related papers (2022-03-27T04:37:54Z) - Density Ratio Estimation via Infinitesimal Classification [85.08255198145304]
We propose DRE-infty, a divide-and-conquer approach to reduce Density ratio estimation (DRE) to a series of easier subproblems.
Inspired by Monte Carlo methods, we smoothly interpolate between the two distributions via an infinite continuum of intermediate bridge distributions.
We show that our approach performs well on downstream tasks such as mutual information estimation and energy-based modeling on complex, high-dimensional datasets.
arXiv Detail & Related papers (2021-11-22T06:26:29Z) - Autoregressive Score Matching [113.4502004812927]
We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores)
For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training.
We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
arXiv Detail & Related papers (2020-10-24T07:01:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.