A Geometric Account of Activation Steering through Angle-Norm Decomposition

• URL: http://arxiv.org/abs/2606.06735v2
• Date: Mon, 08 Jun 2026 18:02:18 GMT
• Title: A Geometric Account of Activation Steering through Angle-Norm Decomposition
• Abstract summary: Linear activation steering has gained popularity as a simple and empirically effective way to control language model behavior.<n>We show that steering methods differ in how they couple two geometric effects: changing a token's angular alignment with a concept direction and changing its hidden-state norm.<n>Our results explain why interventions with similar concept-level effects can behave differently, and suggest that activation steering should be parameterized by interpretable angular and radial components.
• Score: 1.4323566945483497
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Linear activation steering has gained popularity as a simple and empirically effective way to control language model behavior. More recently, spherical steering paradigms have been proposed to address limitations of additive interventions, often motivated by the assumption that hidden-state norm does not carry concept-relevant information. In this work, we revisit this assumption through a controlled empirical study designed to disentangle the roles of angular and radial components. We show that steering methods differ mainly in how they couple two geometric effects: changing a token's angular alignment with a concept direction and changing its hidden-state norm. Across seven language models, we find that concepts are represented primarily in angular structure, supporting the motivation for spherical methods, but that norm remains important for the stability and downstream effects of steering. Our results explain why interventions with similar concept-level effects can behave differently, and suggest that activation steering should be parameterized by interpretable angular and radial components of the intervention, rather than by a single additive coefficient that entangles these two effects.

Related papers

• Disentangling Steering Vectors [13.671243729016568]
  We propose a framework to explicitly isolate individual and semantically consistent features from composite directions.<n>Specifically, we pair positive and negative activations and take their differences to generate a set of instance-level steering vectors.<n>We show that our method yields a set of semantically consistent basis vectors whose steering effects are mutually distinguishable.
  arXiv  Detail & Related papers  (2026-09-07T04:45:32Z)
• The Cylindrical Representation Hypothesis for Language Model Steering [57.97381760521523]
  We show that a central axis captures the main difference between concept absence and presence and drives concept generation.<n>We formalize this as the Cylindrical Representation Hypothesis (CRH)<n>Our experiments verify the existence of the cylindrical structure and demonstrate that CRH provides a valid and practical way to interpret model steering behavior in real settings.
  arXiv  Detail & Related papers  (2026-05-03T12:26:13Z)
• Minimizing Collateral Damage in Activation Steering [26.274685653829792]
  Activation steering is a method for controlling Large Language Model (LLM) behavior by intervening in its internal representations to increase the alignment with a specific target feature direction.<n>Standard interventions, such as vector addition, often cause unintended changes in the alignment of activations along other non-target feature directions.<n>In this work, we provide a mathematical formalization of collateral damage and introduce a principled framework that models steering as a constrained optimization problem.
  arXiv  Detail & Related papers  (2026-05-01T23:52:54Z)
• From Weights to Activations: Is Steering the Next Frontier of Adaptation? [24.89959346599616]
  Post-training adaptation of language models is commonly achieved through parameter updates or input-based methods.<n>We argue that steering should be regarded as a form of model adaptation.
  arXiv  Detail & Related papers  (2026-04-15T17:06:25Z)
• What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal [53.189667624047416]
  We show that different steering methodologies leverage functionally interchangeable circuits when applied at the same layer.<n>We show that steering vectors can be sparsified by up to 90-99% while retaining most performance.
  arXiv  Detail & Related papers  (2026-04-09T17:57:14Z)
• Weight Updates as Activation Shifts: A Principled Framework for Steering [54.70188910511715]
  Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices.<n>We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior.<n>This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site.
  arXiv  Detail & Related papers  (2026-02-28T02:50:04Z)
• Spherical Steering: Geometry-Aware Activation Rotation for Language Models [15.078810641141295]
  Inference-time steering has emerged as a promising paradigm for controlling language models (LMs) without the cost of retraining.<n>In this work, we explore Spherical Steering, a training-free primitive that resolves this trade-off through activation rotation.<n>Our method rotates activations along a geodesic toward a target direction, guiding the activation toward the target concept while preserving the integrity of the signal.
  arXiv  Detail & Related papers  (2026-02-09T00:15:47Z)
• Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics [81.80010043113445]
  Local weight fine-tuning, LoRA-based adaptation, and activation-based interventions are studied in isolation.<n>We present a unified view that frames these interventions as dynamic weight updates induced by a control signal.<n>Across methods, we observe a consistent trade-off between preference and utility: stronger control increases preference while predictably reducing utility.
  arXiv  Detail & Related papers  (2026-02-02T17:04:36Z)
• Angular Steering: Behavior Control via Rotation in Activation Space [1.3400719989424488]
  Angular Steering is a novel and flexible method for behavior modulation.<n>It operates by rotating activations within a fixed two-dimensional subspace.<n>It provides continuous, fine-grained control over behaviors such as refusal and compliance.
  arXiv  Detail & Related papers  (2025-10-30T08:23:35Z)
• SteeringSafety: A Systematic Safety Evaluation Framework of Representation Steering in LLMs [42.189660766537536]
  We introduce SteeringSafety, a systematic framework for evaluating representation steering methods across seven safety perspectives spanning 17 datasets.<n>Our framework provides modularized building blocks for state-of-the-art steering methods, enabling unified implementation of DIM, ACE, CAA, PCA, and LAT with recent enhancements like conditional steering.
  arXiv  Detail & Related papers  (2025-09-16T18:36:22Z)
• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)
• Unifying Perplexing Behaviors in Modified BP Attributions through Alignment Perspective [61.5509267439999]
  We present a unified theoretical framework for methods like GBP, RectGrad, LRP, and DTD.<n>We demonstrate that they achieve input alignment by combining the weights of activated neurons.<n>This alignment improves the visualization quality and reduces sensitivity to weight randomization.
  arXiv  Detail & Related papers  (2025-03-14T07:58:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.