Towards Understanding the Robustness of Sparse Autoencoders

• URL: http://arxiv.org/abs/2604.18756v1
• Date: Mon, 20 Apr 2026 19:00:09 GMT
• Title: Towards Understanding the Robustness of Sparse Autoencoders
• Abstract summary: We present a study of integrating pretrained SAEs into transformer residual streams at inference time.<n>SAE-augmented models achieve up to a 5x reduction in jailbreak success rate relative to the undefended baseline.
• Score: 13.16745936025085
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) remain vulnerable to optimization-based jailbreak attacks that exploit internal gradient structure. While Sparse Autoencoders (SAEs) are widely used for interpretability, their robustness implications remain underexplored. We present a study of integrating pretrained SAEs into transformer residual streams at inference time, without modifying model weights or blocking gradients. Across four model families (Gemma, LLaMA, Mistral, Qwen) and two strong white-box attacks (GCG, BEAST) plus three black-box benchmarks, SAE-augmented models achieve up to a 5x reduction in jailbreak success rate relative to the undefended baseline and reduce cross-model attack transferability. Parametric ablations reveal (i) a monotonic dose-response relationship between L0 sparsity and attack success rate, and (ii) a layer-dependent defense-utility tradeoff, where intermediate layers balance robustness and clean performance. These findings are consistent with a representational bottleneck hypothesis: sparse projection reshapes the optimization geometry exploited by jailbreak attacks.

Related papers

• Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles [1.6385815610837167]
  LAMDA is a training framework that transfers language-grounded structure into TSR models without using adversarial examples or adding inference-time overhead.<n>It consistently improves robustness across all attack-backbone-dataset combinations, with gains of up to +12.5 pp under shadow attacks and +13.2 pp under natural-light attacks.
  arXiv  Detail & Related papers  (2026-08-09T17:05:00Z)
• Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of Refusal [3.441021278275805]
  Behavioral alignment in large language models often masks fragile internal safety representations.<n>We study adversarial suffix attacks as a probe of representational alignment.<n>We find that suppressing refusal globally across all layers and positions is more effective than targeting a single layer-position pair.
  arXiv  Detail & Related papers  (2026-07-09T19:21:29Z)
• Reasoning as an Attack Surface: Adaptive Evolutionary CoT Jailbreaks for LLMs [56.53007443197966]
  Large Reasoning Models (LRMs) have demonstrated remarkable capabilities in reasoning and generation tasks.<n>Their explicit chain-of-thought (CoT) mechanism introduces new security risks, making them particularly vulnerable to jailbreak attacks.<n>We propose an adaptive evolutionary CoT jailbreak framework, called AE-CoT, to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-23T10:11:32Z)
• Structured Semantic Cloaking for Jailbreak Attacks on Large Language Models [28.741175254258422]
  We propose Structured Semantic Cloaking (S2C), a novel multi-dimensional jailbreak attack framework.<n>S2C strategically distributes and reshapes semantic cues such that full intent consolidation requires multi-step inference.<n>We evaluate S2C across multiple open-source and proprietary LLMs using HarmBench and JBB-Behaviors.
  arXiv  Detail & Related papers  (2026-03-17T07:20:48Z)
• Jailbreaking LLMs Without Gradients or Priors: Effective and Transferable Attacks [22.52730333160258]
  We introduce RAILS, a framework that operates solely on model logits.<n>By eliminating gradient dependency, RAILS enables cross-tokenizer ensemble attacks.<n> Empirically, RAILS achieves near 100% success rates on multiple open-source models and high black-box attack transferability to closed-source systems like GPT and Gemini.
  arXiv  Detail & Related papers  (2026-01-06T21:14:13Z)
• Crafting Adversarial Inputs for Large Vision-Language Models Using Black-Box Optimization [27.346913473981193]
  We propose a black-box jailbreak attack on Large Vision-Language Models (LVLMs) via Zeroth-Order optimization using Simultaneous Perturbation Approximation (ZO-SPSA)<n>We evaluate ZO-SPSA on three LVLMs, including InstructBLIP, LLaVA and MiniGPT-4, achieving the highest jailbreak success rate of 83.0% on InstructBLIP.<n>These findings underscore the real-world feasibility of black-box jailbreaks and expose critical weaknesses in the safety mechanisms of current LVLMs.
  arXiv  Detail & Related papers  (2026-01-05T02:49:33Z)
• DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Models [50.21378052667732]
  We conduct an in-depth analysis of dLLM vulnerabilities to jailbreak attacks across two distinct dimensions: intra-step and inter-step dynamics.<n>We propose DiffuGuard, a training-free defense framework that addresses vulnerabilities through a dual-stage approach.
  arXiv  Detail & Related papers  (2025-09-29T05:17:10Z)
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs [33.470999703070866]
  Existing approaches to embedding jailbreak triggers suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability.<n>We propose bi-GRPO, a novel RL-based framework tailored explicitly for jailbreak backdoor injection.
  arXiv  Detail & Related papers  (2025-09-24T05:56:41Z)
• Improving LLM Safety Alignment with Dual-Objective Optimization [81.98466438000086]
  Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
  arXiv  Detail & Related papers  (2025-03-05T18:01:05Z)
• Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraints [81.14852921721793]
  This study aims to understand and enhance the transferability of gradient-based jailbreaking methods.<n>We introduce a novel conceptual framework to elucidate transferability and identify superfluous constraints.<n>Our method increases the overall Transfer Attack Success Rate (T-ASR) across a set of target models with varying safety levels from 18.4% to 50.3%.
  arXiv  Detail & Related papers  (2025-02-25T07:47:41Z)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks [87.64278063236847]
  In this work, we propose a unified threat model for the principled comparison of jailbreak attacks.<n>Our threat model checks if a given jailbreak is likely to occur in the distribution of text.<n>We adapt popular attacks to this threat model, and, for the first time, benchmark these attacks on equal footing with it.
  arXiv  Detail & Related papers  (2024-10-21T17:27:01Z)
• h4rm3l: A language for Composable Jailbreak Attack Synthesis [48.5611060845958]
  h4rm3l is a novel approach that addresses the gap with a human-readable domain-specific language.<n>We show that h4rm3l's synthesized attacks are diverse and more successful than existing jailbreak attacks in literature.
  arXiv  Detail & Related papers  (2024-08-09T01:45:39Z)
• Advancing Generalized Transfer Attack with Initialization Derived Bilevel Optimization and Dynamic Sequence Truncation [49.480978190805125]
  Transfer attacks generate significant interest for black-box applications. Existing works essentially directly optimize the single-level objective w.r.t. surrogate model. We propose a bilevel optimization paradigm, which explicitly reforms the nested relationship between the Upper-Level (UL) pseudo-victim attacker and the Lower-Level (LL) surrogate attacker.
  arXiv  Detail & Related papers  (2024-06-04T07:45:27Z)
• Efficient Adversarial Training in LLMs with Continuous Attacks [99.5882845458567]
  Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses. C-AdvIPO is an adversarial variant of IPO that does not require utility data for adversarially robust alignment.
  arXiv  Detail & Related papers  (2024-05-24T14:20:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.