Steer Like the LLM: Activation Steering that Mimics Prompting

• URL: http://arxiv.org/abs/2605.03907v1
• Date: Tue, 05 May 2026 15:59:42 GMT
• Title: Steer Like the LLM: Activation Steering that Mimics Prompting
• Abstract summary: Large language models can be steered at inference time through prompting or activation interventions.<n> activation steering methods often underperform compared to prompt-based approaches.<n>We propose a framework that formulates prompt steering as a form of activation steering.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models can be steered at inference time through prompting or activation interventions, but activation steering methods often underperform compared to prompt-based approaches. We propose a framework that formulates prompt steering as a form of activation steering and investigates whether distilling successful prompt steering behavior into simpler, interpretable models can close this gap. Our analysis reveals that popular activation steering methods are not faithful to the mechanics of prompt steering, which applies strong interventions on some tokens while barely affecting others. Based on these insights, we introduce Prompt Steering Replacement (PSR) models that estimate token-specific steering coefficients from the activations themselves and are trained to imitate prompt-based interventions. Experiments on three steering benchmarks across multiple language models show that PSR models outperform existing activation steering methods, especially when controlling for high-coherence completions, and also compare favorably to prompting on AxBench and persona steering.

Related papers

• When is Your LLM Steerable? [56.656180566692946]
  Activation steering offers a lightweight approach to control language models' behavior at inference time.<n>Finding the regime and boundaries of successful steering typically requires expensive grid searches.<n>We investigate whether steerability can be predicted from the model's internal states at the beginning of the generation process.
  arXiv  Detail & Related papers  (2026-06-10T02:55:34Z)
• Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation [74.17379276939599]
  We show that activation steering can induce broad misalignment, even in the recent Qwen-3.5 series.<n>We characterize properties of AS-induced EM by analyzing key steering-specific factors, including steering magnitude, the low-rank structure of the steering subspace, and the number of epochs during steering-vector construction.
  arXiv  Detail & Related papers  (2026-06-07T15:34:59Z)
• Contextual Linear Activation Steering of Language Models [14.044056042247037]
  Linear activation steering is a powerful approach for eliciting the capabilities of large language models.<n>Existing methods apply a fixed steering strength to all tokens, resulting in inconsistent steering quality across diverse input prompts.<n>We introduce Contextual Linear Activation Steering (CLAS), a method that dynamically adapts linear activation steering to context-dependent steering strengths.
  arXiv  Detail & Related papers  (2026-04-27T16:54:16Z)
• AMPS: Adaptive Modality Preference Steering via Functional Entropy [66.69992693275061]
  We introduce an instance-aware diagnostic metric that quantifies each modality's information contribution and reveals sample-specific susceptibility to steering.<n> Experimental results show that our instance-aware steering outperforms conventional steering in modulating modality preference.
  arXiv  Detail & Related papers  (2026-02-13T02:29:06Z)
• Mechanistic Indicators of Steering Effectiveness in Large Language Models [3.635648354808971]
  Activation-based steering enables Large Language Models to exhibit targeted behaviors by intervening on intermediate activations without retraining.<n>Despite its widespread use, the mechanistic factors that govern when steering succeeds or fails remain poorly understood.<n>We investigate whether the reliability of steering can be diagnosed using internal model signals.
  arXiv  Detail & Related papers  (2026-02-02T06:56:22Z)
• Steering Language Models Before They Speak: Logit-Level Interventions [9.055997973281919]
  We propose a training-free inference-time logit intervention for controllable generation.<n>Our results show that statistically grounded logit steering can achieve large, consistent, and multi-task control gains.
  arXiv  Detail & Related papers  (2026-01-16T03:00:33Z)
• RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering [62.63376387138257]
  We propose a plug-and-play intervention framework that adaptively steers large language models (LLMs) reasoning in activation space.<n>RISER constructs a library of reusable reasoning vectors and employs a lightweight Router to dynamically compose them for each input.<n>The Router is optimized via reinforcement learning under task-level rewards, activating latent cognitive primitives in an emergent and compositional manner.
  arXiv  Detail & Related papers  (2026-01-14T08:04:33Z)
• Prototype-Based Dynamic Steering for Large Language Models [3.90727941420584]
  Prototype-Based Dynamic Steering (PDS) is a test-time method that amplifies large language model (LLM) reasoning without adding or altering instructions.<n>We introduce "reasoning prototypes" by clustering activation differences between Chain-of-Thought (CoT) and neutral prompts.<n>PDS consistently improves accuracy without fine-tuning or prompt engineering.
  arXiv  Detail & Related papers  (2025-10-07T01:34:28Z)
• KV Cache Steering for Controlling Frozen LLMs [80.50365534625438]
  cache steering is a lightweight method for implicit steering of language models.<n>We apply cache steering to induce chain-of-thought reasoning in small language models.
  arXiv  Detail & Related papers  (2025-07-11T17:59:36Z)
• Instruction Following by Boosting Attention of Large Language Models [11.739148611340964]
  latent steering is a lightweight technique that alters internal activations to guide generation.<n>InstABoost boosts the strength of instruction prompting by altering the model's attention during generation.<n>InstABoost demonstrates superior control success compared to both traditional prompting and latent steering.
  arXiv  Detail & Related papers  (2025-06-16T17:42:35Z)
• Improving Instruction-Following in Language Models through Activation Steering [58.876600545898675]
  We derive instruction-specific vector representations from language models and use them to steer models accordingly.<n>We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion.<n>Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation.
  arXiv  Detail & Related papers  (2024-10-15T08:38:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.