Spectral and Temporal Denoising for Differentially Private Optimization

• URL: http://arxiv.org/abs/2505.04468v1
• Date: Wed, 07 May 2025 14:38:58 GMT
• Title: Spectral and Temporal Denoising for Differentially Private Optimization
• Authors: Hyeju Shin, Kyudan Jung, Seongwon Yun, Juyoung Yun,
• Abstract summary: This paper introduces the FFT-Enhanced Kalman Filter (FFTKF), a differentially private optimization method that addresses the challenge of preserving performance in DP-SGD.<n>FFTKF integrates frequency-domain noise shaping with Kalman filtering to enhance gradient quality while preserving $(varepsilon, delta)$-DP guarantees.<n>With a per-iteration complexity of $mathcalO(d log d)$, FFTKF demonstrates improved test accuracy over DP-SGD and DiSK.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: This paper introduces the FFT-Enhanced Kalman Filter (FFTKF), a differentially private optimization method that addresses the challenge of preserving performance in DP-SGD, where added noise typically degrades model utility. FFTKF integrates frequency-domain noise shaping with Kalman filtering to enhance gradient quality while preserving $(\varepsilon, \delta)$-DP guarantees. It employs a high-frequency shaping mask in the Fourier domain to concentrate differential privacy noise in less informative spectral components, preserving low-frequency gradient signals. A scalar-gain Kalman filter with finite-difference Hessian approximation further refines the denoised gradients. With a per-iteration complexity of $\mathcal{O}(d \log d)$, FFTKF demonstrates improved test accuracy over DP-SGD and DiSK across MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets using CNNs, Wide ResNets, and Vision Transformers. Theoretical analysis confirms that FFTKF maintains equivalent privacy guarantees while achieving a tighter privacy-utility trade-off through reduced noise and controlled bias.

Related papers

• DPZV: Elevating the Tradeoff between Privacy and Utility in Zeroth-Order Vertical Federated Learning [9.302691218735406]
  We propose DPZV, the first ZO optimization framework for Vertical Federated Learning (VFL) that achieves tunable differential privacy with performance guarantees.<n>We conduct a comprehensive theoretical analysis showing that DPZV matches the convergence rate of first-order optimization methods while satisfying formal ($epsilon, delta$)-DP guarantees.<n>Experiments on image and language benchmarks demonstrate that DPZV outperforms several baselines in terms of accuracy under a wide range of privacy constraints.
  arXiv  Detail & Related papers  (2025-02-27T22:07:16Z)
• S-Diff: An Anisotropic Diffusion Model for Collaborative Filtering in Spectral Domain [23.22881271027173]
  We propose S-Diff, inspired by graph-based collaborative filtering.<n>S-Diff maps user interaction vectors into the spectral domain and parameterizes diffusion noise to align with graph frequency.<n>This anisotropic diffusion retains significant low-frequency components, preserving a high signal-to-noise ratio.
  arXiv  Detail & Related papers  (2024-12-31T10:54:41Z)
• Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise [60.92029979853314]
  We investigate the roles of gradient normalization and clipping in ensuring the convergence of Gradient Descent (SGD) under heavy-tailed noise. Our work provides the first theoretical evidence demonstrating the benefits of gradient normalization in SGD under heavy-tailed noise. We introduce an accelerated SGD variant incorporating gradient normalization and clipping, further enhancing convergence rates under heavy-tailed noise.
  arXiv  Detail & Related papers  (2024-10-21T22:40:42Z)
• DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction [57.83978915843095]
  This paper introduces DiSK, a novel framework designed to significantly enhance the performance of differentially private gradients.<n>To ensure practicality for large-scale training, we simplify the Kalman filtering process, minimizing its memory and computational demands.
  arXiv  Detail & Related papers  (2024-10-04T19:30:39Z)
• Frequency-aware Feature Fusion for Dense Image Prediction [99.85757278772262]
  We propose Frequency-Aware Feature Fusion (FreqFusion) for dense image prediction tasks. FreqFusion integrates an Adaptive Low-Pass Filter (ALPF) generator, an offset generator, and an Adaptive High-Pass Filter (AHPF) generator. Comprehensive visualization and quantitative analysis demonstrate that FreqFusion effectively improves feature consistency and sharpens object boundaries.
  arXiv  Detail & Related papers  (2024-08-23T07:30:34Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• Securing Distributed SGD against Gradient Leakage Threats [13.979995939926154]
  This paper presents a holistic approach to gradient leakage resilient distributed gradient Descent (SGD) We analyze two types of strategies for privacy-enhanced federated learning: (i) gradient pruning with random selection or low-rank filtering and (ii) gradient perturbation with additive random noise or differential privacy noise. We present a gradient leakage resilient approach to securing distributed SGD in federated learning, with differential privacy controlled noise as the tool.
  arXiv  Detail & Related papers  (2023-05-10T21:39:27Z)
• Amplitude-Varying Perturbation for Balancing Privacy and Utility in Federated Learning [86.08285033925597]
  This paper presents a new DP perturbation mechanism with a time-varying noise amplitude to protect the privacy of federated learning. We derive an online refinement of the series to prevent FL from premature convergence resulting from excessive perturbation noise. The contribution of the new DP mechanism to the convergence and accuracy of privacy-preserving FL is corroborated, compared to the state-of-the-art Gaussian noise mechanism with a persistent noise amplitude.
  arXiv  Detail & Related papers  (2023-03-07T22:52:40Z)
• Neural Network-augmented Kalman Filtering for Robust Online Speech Dereverberation in Noisy Reverberant Environments [13.49645012479288]
  A neural network-augmented algorithm for noise-robust online dereverberation is proposed. The presented framework allows for robust dereverberation on a single-channel noisy reverberant dataset.
  arXiv  Detail & Related papers  (2022-04-06T11:38:04Z)
• Using Kalman Filter The Right Way: Noise Estimation Is Not Optimal [46.556605821252276]
  We show that even a seemingly small violation of KF assumptions can significantly modify the effective noise. We suggest a method to apply gradient-based optimization efficiently to the symmetric and positive-definite (SPD) parameters of KF.
  arXiv  Detail & Related papers  (2021-04-06T08:59:15Z)
• iffDetector: Inference-aware Feature Filtering for Object Detection [70.8678270164057]
  We introduce a generic Inference-aware Feature Filtering (IFF) module that can easily be combined with modern detectors. IFF performs closed-loop optimization by leveraging high-level semantics to enhance the convolutional features. IFF can be fused with CNN-based object detectors in a plug-and-play manner with negligible computational cost overhead.
  arXiv  Detail & Related papers  (2020-06-23T02:57:29Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.