The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Models

• URL: http://arxiv.org/abs/2606.16801v1
• Date: Mon, 15 Jun 2026 14:42:06 GMT
• Title: The Art of Mixology: Mixup-based Obfuscation for Privacy-Preserving Split Learning in Large Language Models
• Abstract summary: MIXGUARD is a privacy-preserving split learning framework for Large Language Models.<n> token-level obfuscation, representation-level obfuscation, and adaptive gradient mechanisms operate jointly to preserve useful learning signals.<n>We conduct experiments on four classification tasks and four text generation tasks across multiple LLM families, model sizes, architectures, and fine-tuning strategies.
• Score: 25.91253902620787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Split learning provides a practical paradigm for resource-constrained users to train Large Language Models (LLMs) by offloading computation-intensive layers to a server while keeping raw data local. However, existing privacy-preserving split learning methods still face a difficult trade-off among utility, privacy, efficiency, and stability. Specifically, these methods often suffer from substantial utility degradation, remain vulnerable to advanced data reconstruction attacks, incur prohibitive computational and communication overhead, or exhibit unstable performance across different tasks. In this paper, we propose MIXGUARD, a novel mixup-based privacy-preserving split learning framework for LLMs. MIXGUARD introduces token-level obfuscation, representation-level obfuscation, and adaptive gradient perturbation mechanisms, which operate jointly to preserve useful learning signals while preventing privacy leakage to the server. Technically, MIXGUARD first constructs a lightweight calibration model on a public dataset to refine the approximated target representation, and then applies this model during privacy-preserving fine-tuning on private data. We conduct extensive experiments on four classification tasks and four text generation tasks across multiple LLM families, model sizes, architectures, and fine-tuning strategies. The results show that MIXGUARD preserves model utility comparable to non-split training baselines, consistently achieves stronger privacy protection than existing split learning defense methods against state-of-the-art data reconstruction attacks, and remains robust under adaptive attack settings.

Related papers

• Privacy-Preserving Split Learning for Federated LLM Fine-Tuning [12.651705155064102]
  Fine-tuning large language models (LLMs) on domain-specific data is essential for downstream adaptation.<n>In many deployments, a participant cannot hold the complete model locally. Split Learning (SL) addresses this by partitioning the model between the participant and a server so that only a small portion runs locally.<n> Federated Learning (FL) enables collaborative training across participants by sharing only model updates instead of raw data.
  arXiv  Detail & Related papers  (2026-09-09T06:48:21Z)
• Deep Leakage with Generative Flow Matching Denoiser [54.05993847488204]
  We introduce a new deep leakage (DL) attack that integrates a generative Flow Matching (FM) prior into the reconstruction process.<n>Our approach consistently outperforms state-of-the-art attacks across pixel-level, perceptual, and feature-based similarity metrics.
  arXiv  Detail & Related papers  (2026-01-21T14:51:01Z)
• Federated Learning Meets LLMs: Feature Extraction From Heterogeneous Clients [0.0]
  Federated learning (FL) enables collaborative model training without sharing raw data.<n>We propose FedLLM-Align, a framework that leverages pre-trained large language models (LLMs) as universal feature extractors.<n>We evaluate FedLLM-Align on coronary heart disease prediction using partitioned datasets with simulated schema divergence.
  arXiv  Detail & Related papers  (2025-09-29T14:06:52Z)
• Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Masking [16.758567358839294]
  Split Federated Learning (SFL) has emerged as an efficient alternative to traditional Federated Learning (FL)<n>We present PM-SFL, a scalable and privacy-preserving SFL framework that incorporates Probabilistic Mask training to add structured randomness without relying on explicit noise.<n>Experiments on image and wireless sensing tasks demonstrate that PM-SFL consistently improves accuracy, communication efficiency, and to privacy attacks.
  arXiv  Detail & Related papers  (2025-09-18T04:28:08Z)
• Private Training & Data Generation by Clustering Embeddings [74.00687214400021]
  Differential privacy (DP) provides a robust framework for protecting individual data.<n>We introduce a novel principled method for DP synthetic image embedding generation.<n> Empirically, a simple two-layer neural network trained on synthetically generated embeddings achieves state-of-the-art (SOTA) classification accuracy.
  arXiv  Detail & Related papers  (2025-06-20T00:17:14Z)
• PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs [56.04036826558497]
  We introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE)<n>By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances.<n>It almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.
  arXiv  Detail & Related papers  (2025-06-03T15:00:18Z)
• Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training [13.680205342714412]
  Large language models (LLMs) have become the backbone of modern natural language processing but pose privacy concerns about leaking sensitive training data.<n>We propose methodname, a lightweight yet effective empirical privacy defense for protecting training data of language models by leveraging token-specific characteristics.
  arXiv  Detail & Related papers  (2025-02-27T03:37:45Z)
• FewFedPIT: Towards Privacy-preserving and Few-shot Federated Instruction Tuning [54.26614091429253]
  Federated instruction tuning (FedIT) is a promising solution, by consolidating collaborative training across multiple data owners. FedIT encounters limitations such as scarcity of instructional data and risk of exposure to training data extraction attacks. We propose FewFedPIT, designed to simultaneously enhance privacy protection and model performance of federated few-shot learning.
  arXiv  Detail & Related papers  (2024-03-10T08:41:22Z)
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs [92.51670143929056]
  Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data. This process might suffer from privacy issues and violations of data protection regulations. We propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals.
  arXiv  Detail & Related papers  (2023-10-31T03:35:59Z)
• Recovering from Privacy-Preserving Masking with Large Language Models [14.828717714653779]
  We use large language models (LLMs) to suggest substitutes of masked tokens. We show that models trained on the obfuscation corpora are able to achieve comparable performance with the ones trained on the original data.
  arXiv  Detail & Related papers  (2023-09-12T16:39:41Z)
• RelaxLoss: Defending Membership Inference Attacks without Losing Utility [68.48117818874155]
  We propose a novel training framework based on a relaxed loss with a more achievable learning target. RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead. Our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs.
  arXiv  Detail & Related papers  (2022-07-12T19:34:47Z)
• Differentially Private Federated Learning with Laplacian Smoothing [72.85272874099644]
  Federated learning aims to protect data privacy by collaboratively learning a model without sharing private data among users. An adversary may still be able to infer the private training data by attacking the released model. Differential privacy provides a statistical protection against such attacks at the price of significantly degrading the accuracy or utility of the trained models.
  arXiv  Detail & Related papers  (2020-05-01T04:28:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.