Contextual Linear Activation Steering of Language Models

• URL: http://arxiv.org/abs/2604.24693v1
• Date: Mon, 27 Apr 2026 16:54:16 GMT
• Title: Contextual Linear Activation Steering of Language Models
• Abstract summary: Linear activation steering is a powerful approach for eliciting the capabilities of large language models.<n>Existing methods apply a fixed steering strength to all tokens, resulting in inconsistent steering quality across diverse input prompts.<n>We introduce Contextual Linear Activation Steering (CLAS), a method that dynamically adapts linear activation steering to context-dependent steering strengths.
• Score: 14.044056042247037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear activation steering is a powerful approach for eliciting the capabilities of large language models and specializing their behavior using limited labeled data. While effective, existing methods often apply a fixed steering strength to all tokens, resulting in inconsistent steering quality across diverse input prompts. In this work, we introduce Contextual Linear Activation Steering (CLAS), a method that dynamically adapts linear activation steering to context-dependent steering strengths. Across eleven steering benchmarks and four model families, it consistently outperforms standard linear activation steering and matches or exceeds the performance of ReFT and LoRA in settings with limited labeled data. We therefore propose CLAS as a scalable, interpretable, and accurate method for specializing and steering large language models.

Related papers

• Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Control [2.752817022620644]
  Inference-time LLM alignment methods offer an alternative to fine-tuning by directly modifying activations during generation.<n>We show that layer-wise dynamics across multiple LLM architectures and scales are well-approximated by locally-linear models.<n>We derive theoretical bounds on setpoint tracking error, enabling formal guarantees on steering performance.
  arXiv  Detail & Related papers  (2026-04-21T03:09:46Z)
• AMPS: Adaptive Modality Preference Steering via Functional Entropy [66.69992693275061]
  We introduce an instance-aware diagnostic metric that quantifies each modality's information contribution and reveals sample-specific susceptibility to steering.<n> Experimental results show that our instance-aware steering outperforms conventional steering in modulating modality preference.
  arXiv  Detail & Related papers  (2026-02-13T02:29:06Z)
• VISPA: Pluralistic Alignment via Automatic Value Selection and Activation [82.8405077104797]
  We introduce VISPA, a training-free pluralistic alignment framework.<n>We show VISPA is performant across all pluralistic alignment modes in healthcare and beyond.
  arXiv  Detail & Related papers  (2026-01-19T06:38:52Z)
• Steering Language Models Before They Speak: Logit-Level Interventions [9.055997973281919]
  We propose a training-free inference-time logit intervention for controllable generation.<n>Our results show that statistically grounded logit steering can achieve large, consistent, and multi-task control gains.
  arXiv  Detail & Related papers  (2026-01-16T03:00:33Z)
• SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models [4.506695482619111]
  This work introduces SteerVLM, a lightweight steering module for Vision-Language Models (VLMs)<n>Our approach learns from the latent embeddings of paired prompts encoding target and converse behaviors to dynamically adjust activations connecting the language modality with image context.<n>Our steering module requires learning parameters equal to 0.14% of the original VLM's size.
  arXiv  Detail & Related papers  (2025-10-30T17:52:39Z)
• HyperSteer: Activation Steering at Scale with Hypernetworks [25.6004576064897]
  HyperSteer is a family of hypernetwork-based architectures which are trained end-to-end to generate steering vectors conditioned on the natural language steering prompts.<n>We show that scaling HyperSteer with thousands of steering prompts exceeds the performance of state-of-the-art activation steering methods.
  arXiv  Detail & Related papers  (2025-06-03T18:32:01Z)
• LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation [102.1527101235251]
  LangTraj is a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenarios.<n>By conditioning on natural language inputs, LangTraj provides flexible and intuitive control over interactive behaviors.<n>LangTraj demonstrates strong performance in realism, language controllability, and language-conditioned safety-critical simulation.
  arXiv  Detail & Related papers  (2025-04-15T17:14:06Z)
• Improving Instruction-Following in Language Models through Activation Steering [58.876600545898675]
  We derive instruction-specific vector representations from language models and use them to steer models accordingly.<n>We demonstrate how this method can enhance model adherence to constraints such as output format, length, and word inclusion.<n>Our findings demonstrate that activation steering offers a practical and scalable approach for fine-grained control in language generation.
  arXiv  Detail & Related papers  (2024-10-15T08:38:20Z)
• InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance [56.184255657175335]
  We develop textbfInferAligner, a novel inference-time alignment method that utilizes cross-model guidance for harmlessness alignment. Experimental results show that our method can be very effectively applied to domain-specific models in finance, medicine, and mathematics. It significantly diminishes the Attack Success Rate (ASR) of both harmful instructions and jailbreak attacks, while maintaining almost unchanged performance in downstream tasks.
  arXiv  Detail & Related papers  (2024-01-20T10:41:03Z)
• Improving Activation Steering in Language Models with Mean-Centring [10.101141087916133]
  We find that taking the average of activations associated with a target dataset, and subtracting the mean of all training activations, results in effective steering vectors. We also apply mean-centring to extract function vectors, more effectively triggering the execution of a range of natural language tasks by a significant margin.
  arXiv  Detail & Related papers  (2023-12-06T18:27:07Z)
• Offline RL for Natural Language Generation with Implicit Language Q Learning [87.76695816348027]
  Large language models can be inconsistent when it comes to completing user specified tasks. We propose a novel RL method, that combines both the flexible utility framework of RL with the ability of supervised learning. In addition to empirically validating ILQL, we present a detailed empirical analysis situations where offline RL can be useful in natural language generation settings.
  arXiv  Detail & Related papers  (2022-06-05T18:38:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.