From Weights to Activations: Is Steering the Next Frontier of Adaptation?

• URL: http://arxiv.org/abs/2604.14090v1
• Date: Wed, 15 Apr 2026 17:06:25 GMT
• Title: From Weights to Activations: Is Steering the Next Frontier of Adaptation?
• Abstract summary: Post-training adaptation of language models is commonly achieved through parameter updates or input-based methods.<n>We argue that steering should be regarded as a form of model adaptation.
• Score: 24.89959346599616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-training adaptation of language models is commonly achieved through parameter updates or input-based methods such as fine-tuning, parameter-efficient adaptation, and prompting. In parallel, a growing body of work modifies internal activations at inference time to influence model behavior, an approach known as steering. Despite increasing use, steering is rarely analyzed within the same conceptual framework as established adaptation methods. In this work, we argue that steering should be regarded as a form of model adaptation. We introduce a set of functional criteria for adaptation methods and use them to compare steering approaches with classical alternatives. This analysis positions steering as a distinct adaptation paradigm based on targeted interventions in activation space, enabling local and reversible behavioral change without parameter updates. The resulting framing clarifies how steering relates to existing methods, motivating a unified taxonomy for model adaptation.

Related papers

• GEOSTEER: Geodesic Optimization for Activation Steering in Large Language Models [9.438320824151157]
  We propose GeoSteer, an optimization-based method for norm-preserving activation steering.<n>We show that GeoSteer consistently improves over state-of-the-art activation steering baselines.<n>These results suggest that norm-preserving steering can be made more effective by replacing predefined one-step edits with adaptive, geometry-aware optimization.
  arXiv  Detail & Related papers  (2026-09-09T16:55:52Z)
• Looking in the Mirror: Introspecting Side-Effect Misalignments Induced by Fine-Tuning [17.810096531720927]
  Fine-tuning enables a source model to acquire desired capabilities and behaviors in a target domain.<n>Fine-tuning can also degrade alignment properties that were present in the source model.<n>Introspection adapters describe behavioral changes induced by fine-tuning.
  arXiv  Detail & Related papers  (2026-08-05T01:47:07Z)
• Localized Adaptation Reveals Distinct Learning Signatures in Transformers [1.4824891788575416]
  Transformer adaptation is typically distributed across model depth, even when the intended change is narrow.<n>We investigate how adaptation site shapes what a model learns, how well that learning generalizes, and how selectively it is applied.
  arXiv  Detail & Related papers  (2026-07-28T12:49:55Z)
• Weight Updates as Activation Shifts: A Principled Framework for Steering [54.70188910511715]
  Activation steering promises to be an extremely parameter-efficient form of adaptation, but its effectiveness depends on critical design choices.<n>We establish a first-order equivalence between activation-space interventions and weight-space updates, deriving the conditions under which activation steering can replicate fine-tuning behavior.<n>This equivalence yields a principled framework for steering design and identifies the post-block output as a theoretically-backed and highly expressive intervention site.
  arXiv  Detail & Related papers  (2026-02-28T02:50:04Z)
• Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics [81.80010043113445]
  Local weight fine-tuning, LoRA-based adaptation, and activation-based interventions are studied in isolation.<n>We present a unified view that frames these interventions as dynamic weight updates induced by a control signal.<n>Across methods, we observe a consistent trade-off between preference and utility: stronger control increases preference while predictably reducing utility.
  arXiv  Detail & Related papers  (2026-02-02T17:04:36Z)
• Vision-Conditioned Variational Bayesian Last Layer Dynamics Models [2.936118677102837]
  Agile control of robotic systems often requires anticipating how the environment affects system behavior.<n>Traditional modeling approaches often struggle to capture abrupt variations in system behavior.<n>We propose a vision-conditioned variational Bayesian last-layer dynamics model that leverages visual context to anticipate changes in the environment.
  arXiv  Detail & Related papers  (2026-01-14T05:25:18Z)
• Dynamically Scaled Activation Steering [3.177576903071419]
  We introduce Dynamically Scaled Activation Steering (DSAS), a method-agnostic steering framework that decouples when to steer from how to steer.<n>DSAS adaptively modulates the strength of existing steering transformations across layers and inputs, intervening strongly only when undesired behavior is detected.
  arXiv  Detail & Related papers  (2025-12-03T10:50:15Z)
• KV Cache Steering for Controlling Frozen LLMs [80.50365534625438]
  cache steering is a lightweight method for implicit steering of language models.<n>We apply cache steering to induce chain-of-thought reasoning in small language models.
  arXiv  Detail & Related papers  (2025-07-11T17:59:36Z)
• Sparse autoencoders reveal selective remapping of visual concepts during adaptation [54.82630842681845]
  Adapting foundation models for specific purposes has become a standard approach to build machine learning systems.<n>We develop a new Sparse Autoencoder (SAE) for the CLIP vision transformer, named PatchSAE, to extract interpretable concepts.
  arXiv  Detail & Related papers  (2024-12-06T18:59:51Z)
• Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models [88.47454470043552]
  We consider the problem of online fine tuning the parameters of a language model at test time, also known as dynamic evaluation. Online adaptation turns parameters into temporally changing states and provides a form of context-length extension with memory in weights.
  arXiv  Detail & Related papers  (2024-03-03T14:03:48Z)
• Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment [88.06408322210025]
  We study the problem of adapting on-the-fly to novel scenarios during deployment.<n>Our approach, RObust Autonomous Modulation (ROAM), introduces a mechanism based on the perceived value of pre-trained behaviors.<n>We demonstrate that ROAM enables a robot to adapt rapidly to changes in dynamics both in simulation and on a real Go1 quadruped.
  arXiv  Detail & Related papers  (2023-11-02T08:22:28Z)
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments [3.5309638744466167]
  This work developed a meta-learning approach that adapts the control policy on the fly to different changing conditions for robust locomotion. The proposed method constantly updates the interaction model, samples feasible sequences of actions of estimated the state-action trajectories, and then applies the optimal actions to maximize the reward.
  arXiv  Detail & Related papers  (2021-01-19T12:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.