DP-SAPF: Saliency-Aware Parameter Fine-tuning of Public Models for Differentially Private Image Synthesis

• URL: http://arxiv.org/abs/2605.30312v1
• Date: Thu, 28 May 2026 17:53:30 GMT
• Title: DP-SAPF: Saliency-Aware Parameter Fine-tuning of Public Models for Differentially Private Image Synthesis
• Abstract summary: Differentially private (DP) image generates images that preserve statistical characteristics of sensitive dataset.<n>Existing methods fine-tune public models using DP Gradient Descent (DP-SGD) on sensitive images to generate synthetic images.<n>We propose DP-SAPF, which uses a saliency-aware strategy to identify specific target parameters for LoRA training under DP.
• Score: 18.158096061348157
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentially private (DP) image synthesis generates images that preserve the statistical characteristics of a sensitive dataset, enabling sensitive data analysis and usage while providing rigorous guarantees of privacy leakage. Existing methods fine-tune public models using DP Stochastic Gradient Descent (DP-SGD) on sensitive images to generate synthetic images. But full fine-tuning public models on sensitive images is computationally expensive, because current public models typically contain a large number of parameters. Recent work proposes heuristically using Low-Rank Adaptation (LoRA) on all attention-layer parameters of public models to reduce the number of trainable parameters. However, we argue that exhaustive LoRA coverage across all attention-layer parameters is suboptimal in a DP setting, as it leads to noise accumulation and collapse during private training. To address this issue, we propose DP-SAPF, which uses a saliency-aware strategy to identify specific target parameters for LoRA training under DP. DP-SAPF is inspired by the fact that larger gradients signify higher saliency, indicating that these parameters are most critical for the DP learning. Specifically, we feed the sensitive images into public models, compute gradients, and add noise to the gradients to satisfy DP. Then, DP-SAPF identifies the most salient parameters, those exhibiting high gradient magnitudes on sensitive images, for DP fine-tuning. Experiments on four sensitive image datasets show that DP-SAPF improves the utility and fidelity of synthetic images while requiring fewer computational resources than fine-tuning methods without parameter selection.

Related papers

• DP-SelFT: Differentially Private Selective Fine-Tuning for Large Language Models [28.62057098410814]
  We propose DP-SelFT, a framework for differentially private selective fine-tuning of large language models (LLMs)<n> DP-SelFT addresses three DP-specific challenges in parameter selection: avoiding repeated privacy cost, improving stability under noisy estimates, and selecting parameters that remain useful under clipped and noisy updates.<n> Experiments show that DP-SelFT consistently improves the privacy--utility trade-off over existing DP fine-tuning baselines under the same privacy guarantees.
  arXiv  Detail & Related papers  (2026-05-17T12:55:11Z)
• Decomposing Private Image Generation via Coarse-to-Fine Wavelet Modeling [25.30412582520304]
  We propose a spectral DP framework based on the hypothesis that the most privacy-sensitive portions of an image are often low-frequency components in the wavelet space.<n>By restricting the privacy budget to the global structures of the image in the first stage, and leveraging the post-processing property of DP for detail refinement, we achieve promising trade-offs between privacy and utility.
  arXiv  Detail & Related papers  (2026-02-26T17:36:48Z)
• DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuning [51.35628297101575]
  Differentially private (DP) synthetic data generation plays a pivotal role in developing large language models (LLMs) on private data.<n>We introduce Differentially Private Reinforcement Fine-Tuning (DP-RFT), an online reinforcement learning algorithm for synthetic data generation with LLMs.<n>We evaluate DP-RFT for long-form and domain-specific synthetic data generation, such as news articles, meeting transcripts, and medical article abstracts.
  arXiv  Detail & Related papers  (2026-02-20T22:03:56Z)
• VP-NTK: Exploring the Benefits of Visual Prompting in Differentially Private Data Synthesis [48.75967507528161]
  Differentially private (DP) synthetic data has become the de facto standard for releasing sensitive data.<n>One of the emerging techniques in parameter efficient fine-tuning (PEFT) is visual prompting (VP)<n>We show that VP in conjunction with DP-NTK, a DP generator that exploits the power of the neural tangent kernel (NTK) in training DP generative models, achieves a significant performance boost.
  arXiv  Detail & Related papers  (2025-03-20T14:42:11Z)
• DPImageBench: A Unified Benchmark for Differentially Private Image Synthesis [18.158096061348157]
  Differentially private (DP) image synthesis aims to generate artificial images that retain the properties of sensitive images while protecting the privacy of individual images within the dataset.<n>Despite recent advancements, inconsistent--and sometimes flawed--evaluation protocols have been applied across studies.<n>This paper introduces DPImageBench for DP image synthesis, with thoughtful design across several dimensions.
  arXiv  Detail & Related papers  (2025-03-18T19:37:35Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• DPAF: Image Synthesis via Differentially Private Aggregation in Forward Phase [14.76128148793876]
  DPAF is an effective differentially private generative model for high-dimensional image synthesis. It reduces information loss in clipping gradient and low sensitivity for the aggregation. It also tackles the problem of setting a proper batch size by proposing a novel training strategy that asymmetrically trains different parts of the discriminator.
  arXiv  Detail & Related papers  (2023-04-20T16:32:02Z)
• Private Ad Modeling with DP-SGD [58.670969449674395]
  A well-known algorithm in privacy-preserving ML is differentially private gradient descent (DP-SGD) In this work we apply DP-SGD to several ad modeling tasks including predicting click-through rates, conversion rates, and number of conversion events. Our work is the first to empirically demonstrate that DP-SGD can provide both privacy and utility for ad modeling tasks.
  arXiv  Detail & Related papers  (2022-11-21T22:51:16Z)
• Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger [39.93710312222771]
  Per-example clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. We propose an easy-to-use replacement, called automatic clipping, that eliminates the need to tune R for any DPs.
  arXiv  Detail & Related papers  (2022-06-14T19:49:44Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• Smoothed Differential Privacy [55.415581832037084]
  Differential privacy (DP) is a widely-accepted and widely-applied notion of privacy based on worst-case analysis. In this paper, we propose a natural extension of DP following the worst average-case idea behind the celebrated smoothed analysis. We prove that any discrete mechanism with sampling procedures is more private than what DP predicts, while many continuous mechanisms with sampling procedures are still non-private under smoothed DP.
  arXiv  Detail & Related papers  (2021-07-04T06:55:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.