Differentially Private Gradient Flow based on the Sliced Wasserstein Distance

• URL: http://arxiv.org/abs/2312.08227v2
• Date: Mon, 29 Jul 2024 14:50:46 GMT
• Title: Differentially Private Gradient Flow based on the Sliced Wasserstein Distance
• Authors: Ilana Sebag, Muni Sreenivas Pydi, Jean-Yves Franceschi, Alain Rakotomamonjy, Mike Gartrell, Jamal Atif, Alexandre Allauzen,
• Abstract summary: We introduce a novel differentially private generative modeling approach based on a gradient flow in the space of probability measures. Experiments show that our proposed model can generate higher-fidelity data at a low privacy budget.
• Score: 59.1056830438845
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Safeguarding privacy in sensitive training data is paramount, particularly in the context of generative modeling. This can be achieved through either differentially private stochastic gradient descent or a differentially private metric for training models or generators. In this paper, we introduce a novel differentially private generative modeling approach based on a gradient flow in the space of probability measures. To this end, we define the gradient flow of the Gaussian-smoothed Sliced Wasserstein Distance, including the associated stochastic differential equation (SDE). By discretizing and defining a numerical scheme for solving this SDE, we demonstrate the link between smoothing and differential privacy based on a Gaussian mechanism, due to a specific form of the SDE's drift term. We then analyze the differential privacy guarantee of our gradient flow, which accounts for both the smoothing and the Wiener process introduced by the SDE itself. Experiments show that our proposed model can generate higher-fidelity data at a low privacy budget compared to a generator-based model, offering a promising alternative.

Related papers

• Schr\"odinger bridge based deep conditional generative learning [0.0]
  We introduce a novel Schr"odinger bridge based deep generative method for learning conditional distributions. We apply our method to both low-dimensional and high-dimensional conditional generation problems.
  arXiv  Detail & Related papers  (2024-09-25T19:08:13Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Noise in the reverse process improves the approximation capabilities of diffusion models [27.65800389807353]
  In Score based Generative Modeling (SGMs), the state-of-the-art in generative modeling, reverse processes are known to perform better than their deterministic counterparts. This paper delves into the heart of this phenomenon, comparing neural ordinary differential equations (ODEs) and neural dimension equations (SDEs) as reverse processes. We analyze the ability of neural SDEs to approximate trajectories of the Fokker-Planck equation, revealing the advantages of neurality.
  arXiv  Detail & Related papers  (2023-12-13T02:39:10Z)
• Differentiating Metropolis-Hastings to Optimize Intractable Densities [51.16801956665228]
  We develop an algorithm for automatic differentiation of Metropolis-Hastings samplers. We apply gradient-based optimization to objectives expressed as expectations over intractable target densities.
  arXiv  Detail & Related papers  (2023-06-13T17:56:02Z)
• High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance [16.652085114513273]
  We derive high-dimensional scaling limits and fluctuations for the online least-squares Gradient Descent (SGD) algorithm. Our results have several applications, including characterization of the limiting mean-square estimation or prediction errors and their fluctuations.
  arXiv  Detail & Related papers  (2023-04-03T03:50:00Z)
• Score-based Generative Modeling of Graphs via the System of Stochastic Differential Equations [57.15855198512551]
  We propose a novel score-based generative model for graphs with a continuous-time framework. We show that our method is able to generate molecules that lie close to the training distribution yet do not violate the chemical valency rule.
  arXiv  Detail & Related papers  (2022-02-05T08:21:04Z)
• Differentially Private Sliced Wasserstein Distance [5.330240017302619]
  We take the perspective of computing the divergences between distributions under the Differential Privacy (DP) framework. Instead of resorting to the popular gradient-based sanitization method for DP, we tackle the problem at its roots by focusing on the Sliced Wasserstein Distance.
  arXiv  Detail & Related papers  (2021-07-05T08:06:02Z)
• Score-Based Generative Modeling through Stochastic Differential Equations [114.39209003111723]
  We present a differential equation that transforms a complex data distribution to a known prior distribution by injecting noise. A corresponding reverse-time SDE transforms the prior distribution back into the data distribution by slowly removing the noise. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks. We demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.
  arXiv  Detail & Related papers  (2020-11-26T19:39:10Z)
• Identifying Latent Stochastic Differential Equations [29.103393300261587]
  We present a method for learning latent differential equations (SDEs) from high-dimensional time series data. The proposed method learns the mapping from ambient to latent space, and the underlying SDE coefficients, through a self-supervised learning approach. We validate the method through several simulated video processing tasks, where the underlying SDE is known, and through real world datasets.
  arXiv  Detail & Related papers  (2020-07-12T19:46:31Z)
• Stochastic Normalizing Flows [52.92110730286403]
  We introduce normalizing flows for maximum likelihood estimation and variational inference (VI) using differential equations (SDEs) Using the theory of rough paths, the underlying Brownian motion is treated as a latent variable and approximated, enabling efficient training of neural SDEs. These SDEs can be used for constructing efficient chains to sample from the underlying distribution of a given dataset.
  arXiv  Detail & Related papers  (2020-02-21T20:47:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.