Defense Against Prompt Inversion Attacks: An Information-Theoretic Approach for LLM Collaborative Inference

• URL: http://arxiv.org/abs/2606.11592v1
• Date: Wed, 10 Jun 2026 02:36:26 GMT
• Title: Defense Against Prompt Inversion Attacks: An Information-Theoretic Approach for LLM Collaborative Inference
• Abstract summary: Collaborative edge-cloud inference enables resource-constrained devices to leverage large language models (LLMs) by offloading partial computation to cloud servers.<n> transmitting intermediate activations exposes sensitive user prompts to prompt inversion attacks.<n>We propose an information-theoretic defense framework for prompt inversion in collaborative LLM inference.
• Score: 5.218013327284395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Collaborative edge-cloud inference enables resource-constrained devices to leverage large language models (LLMs) by offloading partial computation to cloud servers. However, transmitting intermediate activations exposes sensitive user prompts to prompt inversion attacks, where an adversary reconstructs the original input from shared representations. Existing defenses rely largely on heuristic perturbations or empirical tuning, offering limited theoretical understanding of privacy leakage and its interaction with utility and latency constraints. We propose an information-theoretic defense framework for prompt inversion in collaborative LLM inference. Our approach learns privacy-preserving representations by explicitly minimizing the mutual information between intermediate activations and the input prompt while maintaining task utility under computational constraints. We derive theoretical guarantees on prompt reconstruction error, characterize fundamental privacy-utility tradeoffs, and establish token-level accuracy bounds for downstream inference. We then propose a novel defense based on privacy adapters implemented via low-dimensional information bottlenecks. Extensive experiments across multiple settings demonstrate that our method achieves superior privacy-utility-latency tradeoffs compared to existing defenses (up to 35% reduction in attack success), providing a principled foundation for private and efficient collaborative LLM inference.

Related papers

• What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference [13.194874392306085]
  We introduce ActInv, which solves an intermediate activation matching problem to reconstruct the client's input.<n>To understand this vulnerability, we develop Perturbation Amplification Factor (PAF), a metric for quantifying a layer's inherent resistance to reconstruction.<n>Our analysis reveals that privacy vulnerability is not uniform across layers, with some layers being highly susceptible to leakage while others offer natural resistance.
  arXiv  Detail & Related papers  (2026-05-22T02:14:16Z)
• TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning [60.68349524623048]
  Decompositional jailbreaks pose a critical threat to large language models.<n>We present TwinGate, a stateful dual-encoder defense framework.<n>We construct a comprehensive dataset of over 3.62 million instructions spanning 8,600 distinct malicious intents.
  arXiv  Detail & Related papers  (2026-04-30T13:44:01Z)
• NeuroFilter: Privacy Guardrails for Conversational LLM Agents [50.75206727081996]
  This work addresses the computational challenge of enforcing privacy for agentic Large Language Models (LLMs)<n>NeuroFilter is a guardrail framework that operationalizes contextual integrity by mapping norm violations to simple directions in the model's activation space.<n>A comprehensive evaluation across over 150,000 interactions, covering models from 7B to 70B parameters, illustrates the strong performance of NeuroFilter.
  arXiv  Detail & Related papers  (2026-01-21T05:16:50Z)
• Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuning [24.914116986408327]
  We propose PromptMIA, a membership inference attack tailored to federated prompt-tuning.<n>We empirically show that PromptMIA consistently attains high advantage in this game across diverse benchmark datasets.<n>The results highlight non-trivial challenges for current defenses and offer insights into their limitations.
  arXiv  Detail & Related papers  (2026-01-10T17:50:05Z)
• CLOAK: Contrastive Guidance for Latent Diffusion-Based Data Obfuscation [6.2048266436284605]
  Data obfuscation is a promising technique for mitigating attribute inference attacks by semi-trusted parties with access to time-series data emitted by sensors.<n>We propose Cloak, a novel data obfuscation framework based on latent diffusion models.<n>In contrast to prior work, we employ contrastive learning to extract disentangled representations, which guide the latent diffusion process to retain useful information while concealing private information.
  arXiv  Detail & Related papers  (2025-12-12T23:30:43Z)
• Federated Attention: A Distributed Paradigm for Collaborative LLM Inference over Edge Networks [63.541114376141735]
  Large language models (LLMs) are proliferating rapidly at the edge, delivering intelligent capabilities across diverse application scenarios.<n>However, their practical deployment in collaborative scenarios confronts fundamental challenges: privacy vulnerabilities, communication overhead, and computational bottlenecks.<n>We propose Federated Attention (FedAttn), which integrates the federated paradigm into the self-attention mechanism.
  arXiv  Detail & Related papers  (2025-11-04T15:14:58Z)
• The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaboration [72.33801123508145]
  Large language models (LLMs) are integral to multi-agent systems.<n>Privacy risks emerge that extend beyond memorization, direct inference, or single-turn evaluations.<n>In particular, seemingly innocuous responses, when composed across interactions, can cumulatively enable adversaries to recover sensitive information.
  arXiv  Detail & Related papers  (2025-09-16T16:57:25Z)
• How Breakable Is Privacy: Probing and Resisting Model Inversion Attacks in Collaborative Inference [13.453033795109155]
  Collaborative inference improves computational efficiency for edge devices by transmitting intermediate features to cloud models.<n>There is no established criterion for assessing the difficulty of model inversion attacks (MIAs)<n>We propose the first theoretical criterion to assess MIA difficulty in CI, identifying mutual information, entropy, and effective information volume as key influencing factors.
  arXiv  Detail & Related papers  (2025-01-01T13:00:01Z)
• PATROL: Privacy-Oriented Pruning for Collaborative Inference Against Model Inversion Attacks [15.257413246220032]
  Collaborative inference is a promising solution to enable resource-constrained edge devices to perform inference using state-of-the-art deep neural networks (DNNs) Recent research indicates model inversion attacks (MIAs) can reconstruct input data from intermediate results, posing serious privacy concerns for collaborative inference. This paper provides a viable solution, named PATROL, which develops privacy-oriented pruning to balance privacy, efficiency, and utility of collaborative inference.
  arXiv  Detail & Related papers  (2023-07-20T16:09:07Z)
• Over-the-Air Federated Learning with Privacy Protection via Correlated Additive Perturbations [57.20885629270732]
  We consider privacy aspects of wireless federated learning with Over-the-Air (OtA) transmission of gradient updates from multiple users/agents to an edge server. Traditional perturbation-based methods provide privacy protection while sacrificing the training accuracy. In this work, we aim at minimizing privacy leakage to the adversary and the degradation of model accuracy at the edge server.
  arXiv  Detail & Related papers  (2022-10-05T13:13:35Z)
• Is Vertical Logistic Regression Privacy-Preserving? A Comprehensive Privacy Analysis and Beyond [57.10914865054868]
  We consider vertical logistic regression (VLR) trained with mini-batch descent gradient. We provide a comprehensive and rigorous privacy analysis of VLR in a class of open-source Federated Learning frameworks.
  arXiv  Detail & Related papers  (2022-07-19T05:47:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.