Cocoon: A System Architecture for Differentially Private Training with Correlated Noises

• URL: http://arxiv.org/abs/2510.07304v1
• Date: Wed, 08 Oct 2025 17:56:30 GMT
• Title: Cocoon: A System Architecture for Differentially Private Training with Correlated Noises
• Authors: Donghwan Kim, Xin Gu, Jinho Baek, Timothy Lo, Younghoon Min, Kwangsik Shin, Jongryool Kim, Jongse Park, Kiwan Maeng,
• Abstract summary: DP-SGD adds a noise iteration at each training, which degrades the accuracy of the trained model.<n>A new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations.<n>We propose Cocoon, a hardware-software co-designed framework for efficient training with correlated noises.
• Score: 18.01275303626406
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Machine learning (ML) models memorize and leak training data, causing serious privacy issues to data owners. Training algorithms with differential privacy (DP), such as DP-SGD, have been gaining attention as a solution. However, DP-SGD adds a noise at each training iteration, which degrades the accuracy of the trained model. To improve accuracy, a new family of approaches adds carefully designed correlated noises, so that noises cancel out each other across iterations. We performed an extensive characterization study of these new mechanisms, for the first time to the best of our knowledge, and show they incur non-negligible overheads when the model is large or uses large embedding tables. Motivated by the analysis, we propose Cocoon, a hardware-software co-designed framework for efficient training with correlated noises. Cocoon accelerates models with embedding tables through pre-computing and storing correlated noises in a coalesced format (Cocoon-Emb), and supports large models through a custom near-memory processing device (Cocoon-NMP). On a real system with an FPGA-based NMP device prototype, Cocoon improves the performance by 2.33-10.82x(Cocoon-Emb) and 1.55-3.06x (Cocoon-NMP).

Related papers

• DP-$λ$CGD: Efficient Noise Correlation for Differentially Private Model Training [30.807442477789447]
  We propose a new noise correlation strategy that correlates noise only with the immediately preceding iteration and cancels a controlled portion of it.<n>Our method relies on noise regeneration using a pseudorandom noise generator, eliminating the need to store past noise.<n>We show that the computational overhead is minimal and empirically demonstrate improved accuracy over DP-SGD.
  arXiv  Detail & Related papers  (2026-01-29T21:21:34Z)
• Domain-Incremental Continual Learning for Robust and Efficient Keyword Spotting in Resource Constrained Systems [0.0]
  Keywords Spotting systems with small footprint models deployed on edge devices face significant accuracy and robustness challenges.<n>We propose a comprehensive framework for continual learning designed to adapt to new domains while maintaining computational efficiency.<n>The proposed pipeline integrates a dual-input Convolutional Neural Network, utilizing both Mel Frequency Cepstral Coefficients (MFCC) and Mel-spectrogram features.
  arXiv  Detail & Related papers  (2026-01-22T17:59:31Z)
• Optimizing Model Splitting and Device Task Assignment for Deceptive Signal Assisted Private Multi-hop Split Learning [58.620753467152376]
  In our model, several edge devices jointly perform collaborative training, and some eavesdroppers aim to collect the model and data information from devices.<n>To prevent the eavesdroppers from collecting model and data information, a subset of devices can transmit deceptive signals.<n>We propose a soft actor-critic deep reinforcement learning framework with intrinsic curiosity module and cross-attention.
  arXiv  Detail & Related papers  (2025-07-09T22:53:23Z)
• Private Training & Data Generation by Clustering Embeddings [74.00687214400021]
  Differential privacy (DP) provides a robust framework for protecting individual data.<n>We introduce a novel principled method for DP synthetic image embedding generation.<n> Empirically, a simple two-layer neural network trained on synthetically generated embeddings achieves state-of-the-art (SOTA) classification accuracy.
  arXiv  Detail & Related papers  (2025-06-20T00:17:14Z)
• Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training [10.229653770070202]
  Training generative models with differential privacy (DP) typically involves injecting noise into gradient updates or adapting the discriminator's training procedure. We consider the slicing privacy mechanism that injects noise into random low-dimensional projections of the private data. We present a kernel-based estimator for this divergence, circumventing the need for adversarial training.
  arXiv  Detail & Related papers  (2024-10-25T19:32:58Z)
• Impact of Noisy Supervision in Foundation Model Learning [91.56591923244943]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets.<n>We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Sparsity-Preserving Differentially Private Training of Large Embedding Models [67.29926605156788]
  DP-SGD is a training algorithm that combines differential privacy with gradient descent. Applying DP-SGD naively to embedding models can destroy gradient sparsity, leading to reduced training efficiency. We present two new algorithms, DP-FEST and DP-AdaFEST, that preserve gradient sparsity during private training of large embedding models.
  arXiv  Detail & Related papers  (2023-11-14T17:59:51Z)
• Stochastic Coded Federated Learning: Theoretical Analysis and Incentive Mechanism Design [18.675244280002428]
  We propose a novel FL framework named coded federated learning (SCFL) that leverages coded computing techniques. In SCFL, each edge device uploads a privacy-preserving coded dataset to the server, which is generated by adding noise to the projected local dataset. We show that SCFL learns a better model within the given time and achieves a better privacy-performance tradeoff than the baseline methods.
  arXiv  Detail & Related papers  (2022-11-08T09:58:36Z)
• In-situ animal behavior classification using knowledge distillation and fixed-point quantization [6.649514998517633]
  We take a deep and complex convolutional neural network, known as residual neural network (ResNet), as the teacher model. We implement both unquantized and quantized versions of the developed KD-based models on the embedded systems of our purpose-built collar and ear tag devices.
  arXiv  Detail & Related papers  (2022-09-09T06:07:17Z)
• Large Scale Transfer Learning for Differentially Private Image Classification [51.10365553035979]
  Differential Privacy (DP) provides a formal framework for training machine learning models with individual example level privacy. Private training using DP-SGD protects against leakage by injecting noise into individual example gradients. While this result is quite appealing, the computational cost of training large-scale models with DP-SGD is substantially higher than non-private training.
  arXiv  Detail & Related papers  (2022-05-06T01:22:20Z)
• An Efficient DP-SGD Mechanism for Large Scale NLP Models [28.180412581994485]
  Data used to train Natural Language Understanding (NLU) models may contain private information such as addresses or phone numbers. It is desirable that underlying models do not expose private information contained in the training data. Differentially Private Gradient Descent (DP-SGD) has been proposed as a mechanism to build privacy-preserving models.
  arXiv  Detail & Related papers  (2021-07-14T15:23:27Z)
• DeepGMR: Learning Latent Gaussian Mixture Models for Registration [113.74060941036664]
  Point cloud registration is a fundamental problem in 3D computer vision, graphics and robotics. In this paper, we introduce Deep Gaussian Mixture Registration (DeepGMR), the first learning-based registration method. Our proposed method shows favorable performance when compared with state-of-the-art geometry-based and learning-based registration methods.
  arXiv  Detail & Related papers  (2020-08-20T17:25:16Z)
• Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising [36.935465903971014]
  In this paper, we aim at training deep learning models with differential privacy guarantees. Our key technique is to encode gradients to map them to a smaller vector space. We show that our mechanism outperforms the state-of-the-art DPSGD.
  arXiv  Detail & Related papers  (2020-07-22T16:33:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.