Differentially Private Distributed Bayesian Linear Regression with MCMC

• URL: http://arxiv.org/abs/2301.13778v2
• Date: Wed, 7 Jun 2023 09:07:48 GMT
• Title: Differentially Private Distributed Bayesian Linear Regression with MCMC
• Authors: Bar{\i}\c{s} Alparslan, Sinan Y{\i}ld{\i}r{\i}m, \c{S}. \.Ilker Birbil
• Abstract summary: We consider a distributed setting where multiple parties hold parts of the data and share certain summary statistics of their portions in privacy-preserving noise. We develop a novel generative statistical model for privately shared statistics, which exploits a useful distributional relation between the summary statistics of linear regression. We provide numerical results on both real and simulated data, which demonstrate that the proposed algorithms provide well-rounded estimation and prediction.
• Score: 0.966840768820136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a novel Bayesian inference framework for distributed differentially private linear regression. We consider a distributed setting where multiple parties hold parts of the data and share certain summary statistics of their portions in privacy-preserving noise. We develop a novel generative statistical model for privately shared statistics, which exploits a useful distributional relation between the summary statistics of linear regression. Bayesian estimation of the regression coefficients is conducted mainly using Markov chain Monte Carlo algorithms, while we also provide a fast version to perform Bayesian estimation in one iteration. The proposed methods have computational advantages over their competitors. We provide numerical results on both real and simulated data, which demonstrate that the proposed algorithms provide well-rounded estimation and prediction.

Related papers

• Geometry-Aware Instrumental Variable Regression [56.16884466478886]
  We propose a transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings.
  arXiv  Detail & Related papers  (2024-05-19T17:49:33Z)
• Regression-aware Inference with LLMs [52.764328080398805]
  We show that an inference strategy can be sub-optimal for common regression and scoring evaluation metrics. We propose alternate inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses.
  arXiv  Detail & Related papers  (2024-03-07T03:24:34Z)
• Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains [9.429772474335122]
  We focus on scenarios where data distributions vary across multiple segments of the entire population. We propose a two-stage multiply robust estimation method to improve model performance on each individual segment. Our method is designed to be implemented with commonly used off-the-shelf machine learning models.
  arXiv  Detail & Related papers  (2024-02-21T22:01:10Z)
• Conformal inference for regression on Riemannian Manifolds [49.7719149179179]
  We investigate prediction sets for regression scenarios when the response variable, denoted by $Y$, resides in a manifold, and the covariable, denoted by X, lies in Euclidean space. We prove the almost sure convergence of the empirical version of these regions on the manifold to their population counterparts.
  arXiv  Detail & Related papers  (2023-10-12T10:56:25Z)
• Differentially Private Linear Regression with Linked Data [3.9325957466009203]
  Differential privacy, a mathematical notion from computer science, is a rising tool offering robust privacy guarantees. Recent work focuses on developing differentially private versions of individual statistical and machine learning tasks. We present two differentially private algorithms for linear regression with linked data.
  arXiv  Detail & Related papers  (2023-08-01T21:00:19Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• Approximate Bayesian inference from noisy likelihoods with Gaussian process emulated MCMC [0.24275655667345403]
  We model the log-likelihood function using a Gaussian process (GP) The main methodological innovation is to apply this model to emulate the progression that an exact Metropolis-Hastings (MH) sampler would take. The resulting approximate sampler is conceptually simple and sample-efficient.
  arXiv  Detail & Related papers  (2021-04-08T17:38:02Z)
• Differentially private inference via noisy optimization [3.015622397986615]
  We show that robust statistics can be used in conjunction with noisy gradient descent or noisy Newton methods to obtain optimal private estimators. We demonstrate the effectiveness of a bias correction that leads to enhanced small-sample empirical performance in simulations.
  arXiv  Detail & Related papers  (2021-03-19T19:55:55Z)
• Differentially Private Bayesian Inference for Generalized Linear Models [21.942149563564968]
  Generalized linear models (GLMs) such as logistic regression are among the most widely used arms in data analyst's repertoire. We introduce a generic noise-aware DP Bayesian inference method for a GLM, given a noisy sum of summary statistics. We experimentally demonstrate that the posteriors obtained from our model, while adhering to strong privacy guarantees, are close to the non-private posteriors.
  arXiv  Detail & Related papers  (2020-11-01T10:38:22Z)
• Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
  We formulate a method that learns a finite set of statistics from each return distribution via neural networks. Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
  arXiv  Detail & Related papers  (2020-07-24T05:18:17Z)
• Distributed Sketching Methods for Privacy Preserving Regression [54.51566432934556]
  We leverage randomized sketches for reducing the problem dimensions as well as preserving privacy and improving straggler resilience in asynchronous distributed systems. We derive novel approximation guarantees for classical sketching methods and analyze the accuracy of parameter averaging for distributed sketches. We illustrate the performance of distributed sketches in a serverless computing platform with large scale experiments.
  arXiv  Detail & Related papers  (2020-02-16T08:35:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.