GAPS: Dimension-Level Gates for Conditional Activation Steering

• URL: http://arxiv.org/abs/2609.01878v1
• Date: Tue, 01 Sep 2026 21:21:52 GMT
• Title: GAPS: Dimension-Level Gates for Conditional Activation Steering
• Abstract summary: Activation steering suppresses undesired behaviors in language models by adding a steering vector to the hidden state during generation.<n>Recent conditional methods such as CAST and DSAS improve the behavior-capability trade-off by deciding when to intervene, but once active, they apply the full dense vector to all hidden dimensions.<n>We introduce dimension-level conditioning as a complementary axis of selectivity that also decides which neurons to intervene on.
• Score: 5.5832442227189425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Activation steering suppresses undesired behaviors in language models by adding a steering vector to the hidden state during generation. Recent conditional methods such as CAST and DSAS improve the behavior-capability trade-off by deciding when to intervene, but once active, they apply the full dense vector to all hidden dimensions, regardless of whether a neuron carries concept information or already lies in the desired regime. We introduce dimension-level conditioning as a complementary axis of selectivity that also decides which neurons to intervene on. Our method, GAPS (Gated Activation steering via Posterior and Separability), combines two training-free gates: a static separability gate that restricts steering to neurons with statistically reliable concept information (via AUROC), and a dynamic posterior gate that steers a neuron only when its current activation is better explained by the undesired concept under a Gaussian model. The gates add O(D) overhead per token, and they plug into existing conditional methods. On toxicity mitigation (RealToxicityPrompts) and concept removal (OneSeC) with Gemma-3 (4B) and Qwen-3 (1.7B), GAPS consistently matches or improves the Pareto front of its token-level counterparts; under a fixed capability budget, DSAS+GAPS reduces Gemma-3's toxicity rate from 6.52% to 0.48%, versus 3.52% for DSAS alone. Ablations attribute most of the gain to the posterior gate.

Related papers

• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models [62.522141464687195]
  We introduce Action-JND, which extends the principle of just noticeable difference (JND) to embodied perception.<n>A token change is considered tolerable only when the induced action deviation remains within a tolerated margin.<n>The resulting action-tolerance score serves as a plug-and-play criterion for VLA compression paradigms.
  arXiv  Detail & Related papers  (2026-08-21T15:59:37Z)
• A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models [12.373460256583583]
  Activation steering via sparse autoencoders (SAEs) enables behavioral control of large language models without task-specific fine-tuning, but standard methods apply the steering signal at every generated token.<n>We introduce Token Steering (STS), which gates each token independently with probability $p$, and Block Steering (SBS), which gates a leading window once per sequence.<n>Across two model families and two behavioral tasks, steering only 50% of the tokens recovers most of dense-steering effect while preserving fluency, and steering as few as 30% surpasses prompt-based control.
  arXiv  Detail & Related papers  (2026-07-06T20:25:27Z)
• Output Vector Editing for Memorization Mitigation in Large Language Models [68.30351930772788]
  Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks.<n>Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages.<n>We propose output vector editing, a constrained-optimization edit that locates a small set of neurons responsible for a memorized continuation.
  arXiv  Detail & Related papers  (2026-06-17T07:29:18Z)
• Statistically Grounded Sparse-Feature Interventions for Activation-Space Control in Large Language Models [2.464003792743989]
  Activation steering offers a lightweight alternative to fine-tuning for behavioral control of large language models.<n>We introduce a transparent SAE-feature steering pipeline that ranks sparse features through an unweighted Borda consensus.<n>Across three Gemma-family models, four behavioral domains, and 356 layer-strength configurations, the method produces measurable domain-specific shifts.
  arXiv  Detail & Related papers  (2026-06-05T19:27:31Z)
• Targeted Neuron Modulation via Contrastive Pair Search [0.0]
  We show that neuron-level intervention enables reliable behavioral steering without the quality tradeoffs of residual-stream methods.<n>We find that base models contain similar late-layer discrimination structures but steering these neurons produces only content shifts.<n>More broadly, our findings suggest that alignment fine-tuning transforms pre-existing discrimination structure into a sparse, targetable refusal gate.
  arXiv  Detail & Related papers  (2026-05-12T15:47:38Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• Behavioral Steering in a 35B MoE Language Model via SAE-Decoded Probe Vectors: One Agency Axis, Not Five Traits [0.0]
  We train nine sparse autoencoders (SAEs) on the residual stream of Qwen 3.5-35B-A3B.<n>We use them to identify and steer five agentic behavioral traits.
  arXiv  Detail & Related papers  (2026-03-17T10:05:41Z)
• Dynamically Scaled Activation Steering [3.177576903071419]
  We introduce Dynamically Scaled Activation Steering (DSAS), a method-agnostic steering framework that decouples when to steer from how to steer.<n>DSAS adaptively modulates the strength of existing steering transformations across layers and inputs, intervening strongly only when undesired behavior is detected.
  arXiv  Detail & Related papers  (2025-12-03T10:50:15Z)
• Towards Effective, Stealthy, and Persistent Backdoor Attacks Targeting Graph Foundation Models [62.87838888016534]
  Graph Foundation Models (GFMs) are pre-trained on diverse source domains and adapted to unseen targets.<n>Backdoor attacks against GFMs are non-trivial due to three key challenges.<n>We propose GFM-BA, a novel Backdoor Attack model against Graph Foundation Models.
  arXiv  Detail & Related papers  (2025-11-22T08:52:09Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• Normalized Attention Guidance: Universal Negative Guidance for Diffusion Models [57.20761595019967]
  We present Normalized Attention Guidance (NAG), an efficient, training-free mechanism that applies extrapolation in attention space with L1-based normalization and refinement.<n>NAG restores effective negative guidance where CFG collapses while maintaining fidelity.<n>NAG generalizes across architectures (UNet, DiT), sampling regimes (few-step, multi-step), and modalities (image, video)
  arXiv  Detail & Related papers  (2025-05-27T13:30:46Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.