From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models

• URL: http://arxiv.org/abs/2606.14210v1
• Date: Fri, 12 Jun 2026 07:46:58 GMT
• Title: From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
• Abstract summary: Split learning has emerged as a promising paradigm for large language models (LLMs)<n>We unveil novel vulnerabilities of Split-LLM by presenting Patched Model Inversion with Dual-Sided Initialization (PIDI)<n>To counter threats from both sides, we propose the Adapter-based DualGuard with Mutual Information Defense (ADMI)
• Score: 4.941682005405552
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly deployed in privacy-sensitive domains, where users must balance the risk of data exposure through external APIs against the high computational cost of local deployment. Split learning has therefore emerged as a promising paradigm for LLM fine-tuning and inference under limited local resources. However, it introduces new privacy risks. Prior work primarily studies leakage of private input prompts, typically via inversion attacks on intermediate representations, while the potential for sensitive information leakage through generative response outputs remains largely unexplored. In this work, we unveil novel vulnerabilities of Split-LLM by presenting Patched Model Inversion with Dual-Sided Initialization (PIDI), a two-stage attack that simultaneously targets both private input prompts and output responses in Split-LLM settings. It combines dual-sided initialization with a patched inversion strategy to tackle long sequences, substantially outperforming prior inversion methods. To counter threats from both sides, we further propose the Adapter-based DualGuard with Mutual Information Defense (ADMI), which integrates an adapter-based local warmup strategy and mutual information regularization to provide a strong empirical privacy protection with minimal impact on task performance. Extensive experiments across diverse tasks and models demonstrate that ADMI effectively defends against PIDI and other state-of-the-art inversion attacks. Our code is publicly available at https://github.com/FLAIR-THU/VFLAIR-LLM.

Related papers

• Defense Against Prompt Inversion Attacks: An Information-Theoretic Approach for LLM Collaborative Inference [5.218013327284395]
  Collaborative edge-cloud inference enables resource-constrained devices to leverage large language models (LLMs) by offloading partial computation to cloud servers.<n> transmitting intermediate activations exposes sensitive user prompts to prompt inversion attacks.<n>We propose an information-theoretic defense framework for prompt inversion in collaborative LLM inference.
  arXiv  Detail & Related papers  (2026-06-10T02:36:26Z)
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models [55.30555646945055]
  Text-to-Image (T2I) models are vulnerable to semantic leakage.<n>We introduce DeLeaker, a lightweight approach that mitigates leakage by directly intervening on the model's attention maps.<n>SLIM is the first dataset dedicated to semantic leakage.
  arXiv  Detail & Related papers  (2025-10-16T17:39:21Z)
• The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration [72.33801123508145]
  Large language models (LLMs) are integral to multi-agent systems.<n>Privacy risks emerge that extend beyond memorization, direct inference, or single-turn evaluations.<n>In particular, seemingly innocuous responses, when composed across interactions, can cumulatively enable adversaries to recover sensitive information.
  arXiv  Detail & Related papers  (2025-09-16T16:57:25Z)
• Searching for Privacy Risks in LLM Agents via Simulation [61.229785851581504]
  We present a search-based framework that alternates between improving attack and defense strategies through the simulation of privacy-critical agent interactions.<n>We find that attack strategies escalate from direct requests to sophisticated tactics, such as impersonation and consent forgery.<n>The discovered attacks and defenses transfer across diverse scenarios and backbone models, demonstrating strong practical utility for building privacy-aware agents.
  arXiv  Detail & Related papers  (2025-08-14T17:49:09Z)
• Depth Gives a False Sense of Privacy: LLM Internal States Inversion [17.639108495452785]
  Large Language Models (LLMs) are increasingly integrated into daily routines, yet they raise significant privacy and safety concerns.<n>Recent research proposes collaborative inference, which outsources the early-layer inference to ensure data locality.<n>We propose four inversion attacks that significantly improve the semantic similarity and token matching rate of inverted inputs.
  arXiv  Detail & Related papers  (2025-07-22T09:15:11Z)
• MrM: Black-Box Membership Inference Attacks against Multimodal RAG Systems [31.53306157650065]
  Multimodal retrieval-augmented generation (RAG) systems enhance large vision-language models by integrating cross-modal knowledge.<n>These knowledge databases may contain sensitive information that requires privacy protection.<n>MrM is the first black-box MIA framework targeted at multimodal RAG systems.
  arXiv  Detail & Related papers  (2025-06-09T03:48:50Z)
• Defending against Indirect Prompt Injection by Instruction Detection [109.30156975159561]
  InstructDetector is a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks.<n>InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark.
  arXiv  Detail & Related papers  (2025-05-08T13:04:45Z)
• Dual Defense: Enhancing Privacy and Mitigating Poisoning Attacks in Federated Learning [10.102889257118145]
  Federated learning (FL) is inherently susceptible to privacy breaches and poisoning attacks.<n>This paper introduces a Dual Defense Federated learning (DDFed) framework.<n>DDFed boosts privacy protection and mitigates poisoning attacks, without introducing new participant roles or disrupting the existing FL topology.
  arXiv  Detail & Related papers  (2025-02-08T12:28:20Z)
• Model Inversion in Split Learning for Personalized LLMs: New Insights from Information Bottleneck Theory [11.83473842859642]
  This work is the first to identify model inversion attacks in the split learning framework for personalized LLMs.<n>We propose a two-stage attack system in which the first part projects representations into the embedding space, and the second part uses a generative model to recover text from these embeddings.
  arXiv  Detail & Related papers  (2025-01-10T13:47:13Z)
• PriRoAgg: Achieving Robust Model Aggregation with Minimum Privacy Leakage for Federated Learning [49.916365792036636]
  Federated learning (FL) has recently gained significant momentum due to its potential to leverage large-scale distributed user data.<n>The transmitted model updates can potentially leak sensitive user information, and the lack of central control of the local training process leaves the global model susceptible to malicious manipulations on model updates.<n>We develop a general framework PriRoAgg, utilizing Lagrange coded computing and distributed zero-knowledge proof, to execute a wide range of robust aggregation algorithms while satisfying aggregated privacy.
  arXiv  Detail & Related papers  (2024-07-12T03:18:08Z)
• Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shift [104.76588209308666]
  This paper explores backdoor attacks in LVLM instruction tuning across mismatched training and testing domains.<n>We introduce a new evaluation dimension, backdoor domain generalization, to assess attack robustness.<n>We propose a multimodal attribution backdoor attack (MABA) that injects domain-agnostic triggers into critical areas.
  arXiv  Detail & Related papers  (2024-06-27T02:31:03Z)
• Avoid Adversarial Adaption in Federated Learning by Multi-Metric Investigations [55.2480439325792]
  Federated Learning (FL) facilitates decentralized machine learning model training, preserving data privacy, lowering communication costs, and boosting model performance through diversified data sources. FL faces vulnerabilities such as poisoning attacks, undermining model integrity with both untargeted performance degradation and targeted backdoor attacks. We define a new notion of strong adaptive adversaries, capable of adapting to multiple objectives simultaneously. MESAS is the first defense robust against strong adaptive adversaries, effective in real-world data scenarios, with an average overhead of just 24.37 seconds.
  arXiv  Detail & Related papers  (2023-06-06T11:44:42Z)
• Bilateral Dependency Optimization: Defending Against Model-inversion Attacks [61.78426165008083]
  We propose a bilateral dependency optimization (BiDO) strategy to defend against model-inversion attacks. BiDO achieves the state-of-the-art defense performance for a variety of datasets, classifiers, and MI attacks.
  arXiv  Detail & Related papers  (2022-06-11T10:07:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.