Differentially Private Contrastive Learning via Bounding Group-level Contribution

• URL: http://arxiv.org/abs/2604.26467v1
• Date: Wed, 29 Apr 2026 09:25:06 GMT
• Title: Differentially Private Contrastive Learning via Bounding Group-level Contribution
• Abstract summary: Differentially private (DP) contrastive learning aims to learn general-purpose representations from sensitive data.<n>Existing approaches suffer from severe utility degradation due to over-strong inter-sample dependency.<n>We propose DP-GCL, a principled DP contrastive learning framework that limits gradient dependency through bounding group-level contribution.
• Score: 33.81927216429059
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differentially private (DP) contrastive learning aims to learn general-purpose representations from sensitive data, alleviating the privacy leakage concerns of organizations deploying or sharing embedding models trained on private user content. However, existing approaches suffer from severe utility degradation due to the over-strong inter-sample dependency inherent in standard contrastive objectives, where each sample's gradient depends on all other samples in the batch, amplifying the impact of DP noise. In this work, we argue that effective DP contrastive learning requires explicitly reducing such intrinsic inter-sample reliance. To this end, we propose DP-GCL, a principled DP contrastive learning framework that structurally limits gradient dependency through bounding group-level contribution. DP-GCL partitions each batch into small, disjoint groups and restricts available negative samples to within-group samples, thereby localizing gradient influence and reducing sensitivity. To counteract the resulting loss of negative sample diversity, we further introduce intra-group augmentation, which generates additional negative views without increasing privacy cost. Extensive experiments across eight datasets demonstrate that DP-GCL consistently advances the state of the art in both uni-modal and multi-modal contrastive learning under practical privacy budgets: it improves image classification accuracy by 5.6% and image-text retrieval accuracy by 20.1% over existing DP contrastive methods.

Related papers

• SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training [13.525340904948829]
  We introduce SoftAdaClip, a differentially private training method that replaces hard clipping with a smooth, tanh-based transformation.<n>We evaluate SoftAdaClip on various datasets, including MIMIC-III (clinical text), GOSSIS-eICU (structured healthcare), and Adult Income.<n>Our results show that SoftAdaClip reduces subgroup disparities by up to 87% compared to DP-SGD and up to 48% compared to Adaptive-DPSGD.
  arXiv  Detail & Related papers  (2025-10-01T20:38:37Z)
• Harnessing Group-Oriented Consistency Constraints for Semi-Supervised Semantic Segmentation in CdZnTe Semiconductors [71.44213719783703]
  Intra-group Consistency Augmentation Framework (ICAF) developed to label Cadmium Zinc Telluride (CdZnTe) semiconductor images.<n>ICAF consists of two key modules, the View Augmentation Module (VAM) and the View Correction Module (VCM)<n>ICAF achieves a 70.6% mIoU on the CdZnTe dataset using only 2 group-annotated data.
  arXiv  Detail & Related papers  (2025-08-18T09:40:36Z)
• Decoupled Contrastive Learning for Federated Learning [0.0]
  We introduce Decoupled Contrastive Learning for Federated Learning (DCFL)<n>DCFL is a novel framework that decouples the existing contrastive loss into two objectives.<n>Our results show that DCFL achieves stronger alignment between positive samples and greater uniformity between negative samples.
  arXiv  Detail & Related papers  (2025-08-06T01:39:54Z)
• CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective [48.99488315273868]
  We propose a contrastive knowledge distillation framework that achieves sample-wise logit alignment while preserving semantic consistency.<n>Our approach transfers "dark knowledge" through teacher-student contrastive alignment at the sample level.<n>We conduct comprehensive experiments across three benchmark datasets, including the CIFAR-100, ImageNet-1K, and MS COCO datasets.
  arXiv  Detail & Related papers  (2024-04-22T11:52:40Z)
• Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach [62.000948039914135]
  Using Differentially Private Gradient Descent with Gradient Clipping (DPSGD-GC) to ensure Differential Privacy (DP) comes at the cost of model performance degradation. We propose a new error-feedback (EF) DP algorithm as an alternative to DPSGD-GC. We establish an algorithm-specific DP analysis for our proposed algorithm, providing privacy guarantees based on R'enyi DP.
  arXiv  Detail & Related papers  (2023-11-24T17:56:44Z)
• Differentially Private Federated Clustering over Non-IID Data [59.611244450530315]
  clustering clusters (FedC) problem aims to accurately partition unlabeled data samples distributed over massive clients into finite clients under the orchestration of a server. We propose a novel FedC algorithm using differential privacy convergence technique, referred to as DP-Fed, in which partial participation and multiple clients are also considered. Various attributes of the proposed DP-Fed are obtained through theoretical analyses of privacy protection, especially for the case of non-identically and independently distributed (non-i.i.d.) data.
  arXiv  Detail & Related papers  (2023-01-03T05:38:43Z)
• Towards Effective Image Manipulation Detection with Proposal Contrastive Learning [61.5469708038966]
  We propose Proposal Contrastive Learning (PCL) for effective image manipulation detection. Our PCL consists of a two-stream architecture by extracting two types of global features from RGB and noise views respectively. Our PCL can be easily adapted to unlabeled data in practice, which can reduce manual labeling costs and promote more generalizable features.
  arXiv  Detail & Related papers  (2022-10-16T13:30:13Z)
• Differentially Private Estimation of Heterogeneous Causal Effects [9.355532300027727]
  We introduce a general meta-algorithm for estimating conditional average treatment effects (CATE) with differential privacy guarantees. Our meta-algorithm can work with simple, single-stage CATE estimators such as S-learner and more complex multi-stage estimators such as DR and R-learner.
  arXiv  Detail & Related papers  (2022-02-22T17:21:18Z)
• RDP-GAN: A R\'enyi-Differential Privacy based Generative Adversarial Network [75.81653258081435]
  Generative adversarial network (GAN) has attracted increasing attention recently owing to its impressive ability to generate realistic samples with high privacy protection. However, when GANs are applied on sensitive or private training examples, such as medical or financial records, it is still probable to divulge individuals' sensitive and private information. We propose a R'enyi-differentially private-GAN (RDP-GAN), which achieves differential privacy (DP) in a GAN by carefully adding random noises on the value of the loss function during training.
  arXiv  Detail & Related papers  (2020-07-04T09:51:02Z)
• Removing Disparate Impact of Differentially Private Stochastic Gradient Descent on Model Accuracy [18.69118059633505]
  When we enforce differential privacy in machine learning, the utility-privacy trade-off is different w.r.t. each group. In this work, we analyze the inequality in utility loss by differential privacy and propose a modified differentially private gradient descent (DPSGD) Our experimental evaluation shows how group sample size and group clipping bias affect the impact of differential privacy in DPSGD, and how adaptive clipping for each group helps to mitigate the disparate impact caused by differential privacy in DPSGDF.
  arXiv  Detail & Related papers  (2020-03-08T02:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.