On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference

• URL: http://arxiv.org/abs/2605.04901v1
• Date: Wed, 06 May 2026 13:31:15 GMT
• Title: On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference
• Abstract summary: We propose an attack that aligns shuffled activations to a common permutation and exploits them to extract model weights.<n> Experiments on Pythia-70m and GPT-2 demonstrate that the proposed attack can align shuffled activations with mean squared errors ranging from $10-9$ to $10-6$.<n>With a query cost of approximately $1, the adversary can recover model weights with L1-norm differences ranging from $10-4$ to $10-2$ compared to the oracle weights.
• Score: 28.863461307044435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For Transformer models, cryptographically secure inference ensures that the client learns only the final output, while the server learns nothing about the client's input. However, securely computing nonlinear layers remains a major efficiency bottleneck due to the substantial communication rounds and data transmission required. To address this issue, prior works reveal intermediate activations to the client, allowing nonlinear operations to be computed in plaintext. Although this approach significantly improves efficiency, exposing activations enables adversaries to extract model weights. To mitigate this risk, existing works employ a shuffling defense that reveals only randomly permuted activations to the client. In this work, we show that the shuffling defense is not as robust as previously claimed. We propose an attack that aligns differently shuffled activations to a common permutation and subsequently exploits them to extract model weights. Experiments on Pythia-70m and GPT-2 demonstrate that the proposed attack can align shuffled activations with mean squared errors ranging from $10^{-9}$ to $10^{-6}$. With a query cost of approximately \$1, the adversary can recover model weights with L1-norm differences ranging from $10^{-4}$ to $10^{-2}$ compared to the oracle weights.

Related papers

• Scaling Laws for Behavioral Foundation Models over User Event Sequences [2.924581427482972]
  We study a common two-part behavioral-model architecture, a feature-based event embedder, and a decoder-only transformer.<n>Across roughly 600 runs on real interaction data, spanning $1015$-$1019$ training FLOPs, we vary four deployment-relevant axes.<n> Compute-optimal training is data-heavy relative to text at low compute, but its $D/N$ ratio moves toward the Chinchilla as compute increases.
  arXiv  Detail & Related papers  (2026-06-03T15:59:25Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• ZORRO: Zero-Knowledge Robustness and Privacy for Split Learning (Full Version) [58.595691399741646]
  Split Learning (SL) is a distributed learning approach that enables resource-constrained clients to collaboratively train deep neural networks (DNNs)<n>This setup enables SL to leverage server capacities without sharing data, making it highly effective in resource-constrained environments dealing with sensitive data.<n>We present ZORRO, a private, verifiable, and robust SL defense scheme.
  arXiv  Detail & Related papers  (2025-09-11T18:44:09Z)
• BitHydra: Towards Bit-flip Inference Cost Attack against Large Language Models [22.695878922889715]
  We introduce the first bit-flip inference cost attack that directly modifies model weights to induce persistent overhead for all users of a compromised LLM.<n>We instantiate this attack paradigm with BitHydra, which (1) minimizes a loss that suppresses the end-of-sequence token (i.e., EOS) and (2) employs an efficient yet effective critical-bit search focused on the EOS embedding vector.
  arXiv  Detail & Related papers  (2025-05-22T13:36:00Z)
• FLTG: Byzantine-Robust Federated Learning via Angle-Based Defense and Non-IID-Aware Weighting [2.4568855903094815]
  Byzantine attacks threaten training integrity by manipulating malicious clients' updates.<n>Existing methods struggle with limited robustness under high malicious client ratios and sensitivity to non-i.i.d. data.<n>We propose FLTG, a novel aggregation algorithm integrating angle-based defense and dynamic reference selection.
  arXiv  Detail & Related papers  (2025-05-19T08:39:07Z)
• FL-PLAS: Federated Learning with Partial Layer Aggregation for Backdoor Defense Against High-Ratio Malicious Clients [7.1383449614815415]
  Federated learning (FL) is gaining increasing attention as an emerging collaborative machine learning approach.<n>The fundamental algorithm of FL, Federated Averaging (FedAvg), is susceptible to backdoor attacks.<n>We propose a novel defense algorithm, FL-PLAS, which can effectively protect local models from backdoor attacks.
  arXiv  Detail & Related papers  (2025-05-17T14:16:47Z)
• Towards Model Resistant to Transferable Adversarial Examples via Trigger Activation [95.3977252782181]
  Adversarial examples, characterized by imperceptible perturbations, pose significant threats to deep neural networks by misleading their predictions.<n>We introduce a novel training paradigm aimed at enhancing robustness against transferable adversarial examples (TAEs) in a more efficient and effective way.
  arXiv  Detail & Related papers  (2025-04-20T09:07:10Z)
• Pareto-Secure Machine Learning (PSML): Fingerprinting and Securing Inference Serving Systems [0.0]
  Existing black-box attacks assume a single model can be repeatedly selected for serving inference requests. We propose a query-efficient fingerprinting algorithm to enable the attacker to trigger any desired model consistently. We counter the proposed attack with a noise-based defense mechanism that thwarts fingerprinting by adding noise to the specified performance metrics.
  arXiv  Detail & Related papers  (2023-07-03T18:53:47Z)
• G$^2$uardFL: Safeguarding Federated Learning Against Backdoor Attacks through Attributed Client Graph Clustering [116.4277292854053]
  Federated Learning (FL) offers collaborative model training without data sharing. FL is vulnerable to backdoor attacks, where poisoned model weights lead to compromised system integrity. We present G$2$uardFL, a protective framework that reinterprets the identification of malicious clients as an attributed graph clustering problem.
  arXiv  Detail & Related papers  (2023-06-08T07:15:04Z)
• Hindering Adversarial Attacks with Implicit Neural Representations [25.422201099331637]
  Lossy Implicit Network Activation Coding (LINAC) defence successfully hinders several common adversarial attacks. We devise a Parametric Bypass Approximation (PBA) attack strategy for key-based defences, which successfully invalidates an existing method in this category.
  arXiv  Detail & Related papers  (2022-10-22T13:10:24Z)
• Towards Variable-Length Textual Adversarial Attacks [68.27995111870712]
  It is non-trivial to conduct textual adversarial attacks on natural language processing tasks due to the discreteness of data. In this paper, we propose variable-length textual adversarial attacks(VL-Attack) Our method can achieve $33.18$ BLEU score on IWSLT14 German-English translation, achieving an improvement of $1.47$ over the baseline model.
  arXiv  Detail & Related papers  (2021-04-16T14:37:27Z)
• Weight Poisoning Attacks on Pre-trained Models [103.19413805873585]
  We show that it is possible to construct weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose backdoors'' after fine-tuning. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat.
  arXiv  Detail & Related papers  (2020-04-14T16:51:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.