Differentially Private Linear Regression and Synthetic Data Generation with Statistical Guarantees

• URL: http://arxiv.org/abs/2510.16974v1
• Date: Sun, 19 Oct 2025 19:30:41 GMT
• Title: Differentially Private Linear Regression and Synthetic Data Generation with Statistical Guarantees
• Authors: Shurong Lin, Aleksandra Slavković, Deekshith Reddy Bhoomireddy,
• Abstract summary: In social sciences, small- to medium-scale datasets are common and linear regression (LR) is canonical.<n>In privacy-aware settings, much work has focused on differentially private (DP) LR, but mostly on point estimation with limited attention to uncertainty quantification.
• Score: 41.99844472131922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In social sciences, small- to medium-scale datasets are common and linear regression (LR) is canonical. In privacy-aware settings, much work has focused on differentially private (DP) LR, but mostly on point estimation with limited attention to uncertainty quantification. Meanwhile, synthetic data generation (SDG) is increasingly important for reproducibility studies, yet current DP LR methods do not readily support it. Mainstream SDG approaches are either tailored to discretized data, making them less suitable for continuous regression, or rely on deep models that require large datasets, limiting their use for the smaller, continuous data typical in social science. We propose a method for LR with valid inference under Gaussian DP: a DP bias-corrected estimator with asymptotic confidence intervals (CIs) and a general SDG procedure in which regression on the synthetic data matches our DP regression. Our binning-aggregation strategy is effective in small- to moderate-dimensional settings. Experiments show our method (1) improves accuracy over existing methods, (2) provides valid CIs, and (3) produces more reliable synthetic data for downstream ML tasks than current DP SDGs.

Related papers

• Differentially Private Truncation of Unbounded Data via Public Second Moments [4.662174186673445]
  We propose Public-moment-guided Truncation (PMT), which transforms private data using the public second-moment matrix.<n>PMT substantially improves the accuracy and stability of differential privacy models.
  arXiv  Detail & Related papers  (2026-02-25T12:21:30Z)
• Differentially Private Inference for Longitudinal Linear Regression [9.16331221881594]
  We develop a comprehensive framework for estimation and inference in longitudinal linear regression under user-level DP.<n>For inference, we develop a privatized estimator that is automatically heteroskedasticity- and autocorrelation-consistent.<n>These results provide the first unified framework for practical user-level DP estimation and inference.
  arXiv  Detail & Related papers  (2026-01-15T17:47:02Z)
• More Data or Better Algorithms: Latent Diffusion Augmentation for Deep Imbalanced Regression [0.0]
  In many real-world regression tasks, the data distribution is heavily skewed, and models learn predominantly from abundant majority samples.<n>We propose LatentDiff, a novel framework that uses conditional diffusion models with priority-based generation to synthesize high-quality features in the latent representation space.<n>Experiments on three DIR benchmarks demonstrate substantial improvements in minority regions while maintaining overall accuracy.
  arXiv  Detail & Related papers  (2025-09-27T10:44:28Z)
• A Simplified Analysis of SGD for Linear Regression with Weight Averaging [64.2393952273612]
  Recent work bycitetzou 2021benign provides sharp rates for SGD optimization in linear regression using constant learning rate.<n>We provide a simplified analysis recovering the same bias and variance bounds provided incitepzou 2021benign based on simple linear algebra tools.<n>We believe our work makes the analysis of gradient descent on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling.
  arXiv  Detail & Related papers  (2025-06-18T15:10:38Z)
• Better Rates for Private Linear Regression in the Proportional Regime via Aggressive Clipping [19.186034457189162]
  A common approach is to set the clipping constant much larger than the expected norm of the per-sample gradients.<n>While simplifying the analysis, this is however in sharp contrast with what empirical evidence suggests to optimize performance.<n>Our work bridges this gap between theory and practice by crucially operating in a regime where clipping happens frequently.
  arXiv  Detail & Related papers  (2025-05-22T07:34:27Z)
• DispFormer: A Pretrained Transformer Incorporating Physical Constraints for Dispersion Curve Inversion [56.64622091009756]
  This study introduces DispFormer, a transformer-based neural network for $v_s$ profile inversion from Rayleigh-wave phase and group dispersion curves.<n>DispFormer processes dispersion data independently at each period, allowing it to handle varying lengths without requiring network modifications or strict alignment between training and testing datasets.
  arXiv  Detail & Related papers  (2025-01-08T09:08:24Z)
• Adaptive debiased SGD in high-dimensional GLMs with streaming data [4.704144189806667]
  This paper introduces a novel approach to online inference in high-dimensional generalized linear models.<n>Our method operates in a single-pass mode, making it different from existing methods that require full dataset access or large-dimensional summary statistics storage.<n>The core of our methodological innovation lies in an adaptive descent algorithm tailored for dynamic objective functions, coupled with a novel online debiasing procedure.
  arXiv  Detail & Related papers  (2024-05-28T15:36:48Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)
• Federated Latent Class Regression for Hierarchical Data [5.110894308882439]
  Federated Learning (FL) allows a number of agents to participate in training a global machine learning model without disclosing locally stored data. We propose a novel probabilistic model, Hierarchical Latent Class Regression (HLCR), and its extension to Federated Learning, FEDHLCR. Our inference algorithm, being derived from Bayesian theory, provides strong convergence guarantees and good robustness to overfitting. Experimental results show that FEDHLCR offers fast convergence even in non-IID datasets.
  arXiv  Detail & Related papers  (2022-06-22T00:33:04Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• On the Practicality of Differential Privacy in Federated Learning by Tuning Iteration Times [51.61278695776151]
  Federated Learning (FL) is well known for its privacy protection when training machine learning models among distributed clients collaboratively. Recent studies have pointed out that the naive FL is susceptible to gradient leakage attacks. Differential Privacy (DP) emerges as a promising countermeasure to defend against gradient leakage attacks.
  arXiv  Detail & Related papers  (2021-01-11T19:43:12Z)
• Least Squares Regression with Markovian Data: Fundamental Limits and Algorithms [69.45237691598774]
  We study the problem of least squares linear regression where the data-points are dependent and are sampled from a Markov chain. We establish sharp information theoretic minimax lower bounds for this problem in terms of $tau_mathsfmix$. We propose an algorithm based on experience replay--a popular reinforcement learning technique--that achieves a significantly better error rate.
  arXiv  Detail & Related papers  (2020-06-16T04:26:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.