Mechanistic Interpretability in the Presence of Architectural Obfuscation

• URL: http://arxiv.org/abs/2506.18053v1
• Date: Sun, 22 Jun 2025 14:39:16 GMT
• Title: Mechanistic Interpretability in the Presence of Architectural Obfuscation
• Abstract summary: Architectural obfuscation is a lightweight substitute for heavyweight cryptography in privacy-preserving large-language-model (LLM) inference.<n>We analyze a GPT-2-small model trained from scratch with a representative obfuscation map.<n>Our findings reveal that obfuscation dramatically alters activation patterns within attention heads yet preserves the layer-wise computational graph.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Architectural obfuscation - e.g., permuting hidden-state tensors, linearly transforming embedding tables, or remapping tokens - has recently gained traction as a lightweight substitute for heavyweight cryptography in privacy-preserving large-language-model (LLM) inference. While recent work has shown that these techniques can be broken under dedicated reconstruction attacks, their impact on mechanistic interpretability has not been systematically studied. In particular, it remains unclear whether scrambling a network's internal representations truly thwarts efforts to understand how the model works, or simply relocates the same circuits to an unfamiliar coordinate system. We address this gap by analyzing a GPT-2-small model trained from scratch with a representative obfuscation map. Assuming the obfuscation map is private and the original basis is hidden (mirroring an honest-but-curious server), we apply logit-lens attribution, causal path-patching, and attention-head ablation to locate and manipulate known circuits. Our findings reveal that obfuscation dramatically alters activation patterns within attention heads yet preserves the layer-wise computational graph. This disconnect hampers reverse-engineering of user prompts: causal traces lose their alignment with baseline semantics, and token-level logit attributions become too noisy to reconstruct. At the same time, feed-forward and residual pathways remain functionally intact, suggesting that obfuscation degrades fine-grained interpretability without compromising top-level task performance. These results establish quantitative evidence that architectural obfuscation can simultaneously (i) retain global model behaviour and (ii) impede mechanistic analyses of user-specific content. By mapping where interpretability breaks down, our study provides guidance for future privacy defences and for robustness-aware interpretability tooling.

Related papers

• DEFUSE: Generalizable Backdoor Defense for Self-Supervised Encoders with Generative Priors [40.98521902229488]
  Self-supervised learning (SSL) encoders are vulnerable to backdoor attacks.<n>DEFUSE is a generalizable backdoor detection framework for SSL encoders.
  arXiv  Detail & Related papers  (2026-08-26T14:24:05Z)
• Defending against Model Extraction for GNNs with Model Reprogramming [57.82196268649236]
  Graph Neural Networks (GNNs) serve as the backbone for high-stakes applications in Machine-Learning-as-a-Service (ML)<n>Still, their black-box deployment exposes them to Model Extraction (ME) attacks.<n>Existing defenses suffer from a critical ''Euclidean bias''<n>We propose GraphRP, a proactive defense framework that repurposes Model Reprogramming for security.
  arXiv  Detail & Related papers  (2026-08-11T23:20:15Z)
• Beyond Homophily: Towards Generalized Graph Reconstruction Attack and Defense [80.28330585803133]
  Graph neural networks (GNNs) are widely deployed on computation data, yet they can leak sensitive or proprietary information about the training graph adjacency.<n>This work studies graph reconstruction attacks (GRA), a form of model inversion that reconstructs the training adjacency from a trained GNN.<n>We propose MC-GRA (+), which reconstructs the adjacency by optimizing a surrogate adjacency whose GNN-induced representations align with those of the target model.<n>On the defense side, we propose MC-GPB (+), which suppresses adjacency-dependent information throughout the representation chain
  arXiv  Detail & Related papers  (2026-06-06T09:29:27Z)
• Backdoor Channels Hidden in Latent Space: Cryptographic Undetectability in Modern Neural Networks [0.9099663022952497]
  We investigate whether comparable undetectability extends to modern, end-to-end trained networks.<n>We construct such an attack mechanism for state-of-the-art architectures, closely aligned to the cryptographic notion of undetectability.<n>Our results establish that cryptographic backdoors need not be artefacts requiring exotic architectures or artificial constructions, but identifiable as latent properties inherent to the geometry of learned representations.
  arXiv  Detail & Related papers  (2026-05-13T09:06:25Z)
• Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoors [9.503673758168693]
  Local fine-tuning datasets routinely contain sensitive secrets such as API keys, personal identifiers, and financial records.<n>We exploit a practical but overlooked supply-chain vector -- model code camouflaged as standard architectural definitions.<n>We introduce a deterministic full-chain mechanism: it locks onto token-level secrets in dynamic computation flows via online tensor-rule matching.
  arXiv  Detail & Related papers  (2026-04-30T05:03:08Z)
• The Semantic Trap: Do Fine-tuned LLMs Learn Vulnerability Root Cause or Just Functional Pattern? [14.472036099680961]
  We propose TrapEval, a comprehensive evaluation framework designed to disentangle vulnerability root cause from functional pattern.<n>We fine-tune five representative state-of-the-art LLMs across three model families and evaluate them under cross-dataset testing, semantic-preservings, and varying degrees of semantic gap measured by CodeBLEU.<n>Our findings serve as a wake-up call: high benchmark scores on traditional datasets may be illusory, masking the model's inability to understand the true causal logic of vulnerabilities.
  arXiv  Detail & Related papers  (2026-01-30T07:19:17Z)
• The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition [31.827344197678126]
  Tokenizer transplant introduces a supply-chain vulnerability.<n>By exploiting the geometry of coefficient reuse, our attack creates an asymmetric realizability gap.<n> Empirically, the attack is training-free and achieves spectral mimicry to evade outlier detection.
  arXiv  Detail & Related papers  (2025-12-31T19:00:03Z)
• How a Bit Becomes a Story: Semantic Steering via Differentiable Fault Injection [1.690922615975256]
  This work investigates how low-level, bitwise perturbations (fault injection) can influence the semantic meaning of its generated descriptions.<n>In image captioning models, a single flipped bit might subtly alter how visual features map to words, shifting the entire narrative an AI tells about the world.<n>We design a differentiable fault analysis framework, BLADE, that uses gradient-based sensitivity estimation to locate semantically critical bits.
  arXiv  Detail & Related papers  (2025-12-09T04:04:19Z)
• Accuracy is Not Enough: Poisoning Interpretability in Federated Learning via Color Skew [1.732435844754418]
  We show that small color perturbations applied by adversarial clients can shift a model's saliency maps away from semantically meaningful regions.<n>Our attack reduces peak activation overlap in Grad-CAM explanations by up to 35% while preserving classification accuracy above 96% on all evaluated datasets.
  arXiv  Detail & Related papers  (2025-11-17T16:07:54Z)
• From Memorization to Reasoning in the Spectrum of Loss Curvature [6.463682206736737]
  We show that memorization can be disentangled in the weights of both language models (LMs) and vision transformers (ViTs)<n>We analyze the editing procedure extensively on its effect on downstream tasks in LMs, and find that fact retrieval and arithmetic are specifically and consistently negatively affected.
  arXiv  Detail & Related papers  (2025-10-28T10:09:35Z)
• Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer [0.0]
  Chain-of-thought (CoT) reasoning has enabled transformer-based language models to excel at complex mathematics and multi-step planning.<n>In standard decoder-only architectures, these reasoning steps are externalized in natural language, improving interpretability at the cost of efficiency.<n>We investigate whether such reasoning structures emerge in Huginn-3.5B, a depth-recurrent Transformer that reuses layers at inference time without increasing parameter count.
  arXiv  Detail & Related papers  (2025-07-02T23:35:21Z)
• SINDER: Repairing the Singular Defects of DINOv2 [61.98878352956125]
  Vision Transformer models trained on large-scale datasets often exhibit artifacts in the patch token they extract. We propose a novel fine-tuning smooth regularization that rectifies structural deficiencies using only a small dataset.
  arXiv  Detail & Related papers  (2024-07-23T20:34:23Z)
• GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features [68.14842693208465]
  GeneralAD is an anomaly detection framework designed to operate in semantic, near-distribution, and industrial settings. We propose a novel self-supervised anomaly generation module that employs straightforward operations like noise addition and shuffling to patch features. We extensively evaluated our approach on ten datasets, achieving state-of-the-art results in six and on-par performance in the remaining.
  arXiv  Detail & Related papers  (2024-07-17T09:27:41Z)
• Model Pairing Using Embedding Translation for Backdoor Attack Detection on Open-Set Classification Tasks [63.269788236474234]
  We propose to use model pairs on open-set classification tasks for detecting backdoors. We show that this score, can be an indicator for the presence of a backdoor despite models being of different architectures. This technique allows for the detection of backdoors on models designed for open-set classification tasks, which is little studied in the literature.
  arXiv  Detail & Related papers  (2024-02-28T21:29:16Z)
• Spatial-Frequency Discriminability for Revealing Adversarial Perturbations [53.279716307171604]
  Vulnerability of deep neural networks to adversarial perturbations has been widely perceived in the computer vision community. Current algorithms typically detect adversarial patterns through discriminative decomposition for natural and adversarial data. We propose a discriminative detector relying on a spatial-frequency Krawtchouk decomposition.
  arXiv  Detail & Related papers  (2023-05-18T10:18:59Z)
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation [57.92972327649165]
  This work explores a deep learning approach to automatically learn the insecure patterns from code corpora. Because code naturally admits graph structures with parsing, we develop a novel graph neural network (GNN) to exploit both the semantic context and structural regularity of a program.
  arXiv  Detail & Related papers  (2021-09-07T21:24:36Z)
• Where and What? Examining Interpretable Disentangled Representations [96.32813624341833]
  Capturing interpretable variations has long been one of the goals in disentanglement learning. Unlike the independence assumption, interpretability has rarely been exploited to encourage disentanglement in the unsupervised setting. In this paper, we examine the interpretability of disentangled representations by investigating two questions: where to be interpreted and what to be interpreted.
  arXiv  Detail & Related papers  (2021-04-07T11:22:02Z)
• Hidden Backdoor Attack against Semantic Segmentation Models [60.0327238844584]
  The emphbackdoor attack intends to embed hidden backdoors in deep neural networks (DNNs) by poisoning training data. We propose a novel attack paradigm, the emphfine-grained attack, where we treat the target label from the object-level instead of the image-level. Experiments show that the proposed methods can successfully attack semantic segmentation models by poisoning only a small proportion of training data.
  arXiv  Detail & Related papers  (2021-03-06T05:50:29Z)
• Generating Out of Distribution Adversarial Attack using Latent Space Poisoning [5.1314136039587925]
  We propose a novel mechanism of generating adversarial examples where the actual image is not corrupted. latent space representation is utilized to tamper with the inherent structure of the image. As opposed to gradient-based attacks, the latent space poisoning exploits the inclination of classifiers to model the independent and identical distribution of the training dataset.
  arXiv  Detail & Related papers  (2020-12-09T13:05:44Z)
• Backdoor Attacks on the DNN Interpretation System [16.587968446342995]
  Interpretability is crucial to understand the inner workings of deep neural networks (DNNs) We design a backdoor attack that alters the saliency map produced by the network for an input image only with injected trigger. We show that our attacks constitute a serious security threat when deploying deep learning models developed by untrusty sources.
  arXiv  Detail & Related papers  (2020-11-21T01:54:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.