Steering at the Source: Style Modulation Heads for Robust Persona Control

• URL: http://arxiv.org/abs/2603.13249v1
• Date: Tue, 24 Feb 2026 07:26:50 GMT
• Title: Steering at the Source: Style Modulation Heads for Robust Persona Control
• Abstract summary: Activation steering offers a computationally efficient mechanism for controlling Large Language Models (LLMs) without fine-tuning.<n>In this work, we identify a sparse subset of attention heads that independently govern persona and style formation.<n>We demonstrate that intervention targeting only these specific heads achieves robust behavioral control while significantly mitigating the coherency degradation observed in residual stream steering.
• Score: 1.842570462253593
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Activation steering offers a computationally efficient mechanism for controlling Large Language Models (LLMs) without fine-tuning. While effectively controlling target traits (e.g., persona), coherency degradation remains a major obstacle to safety and practical deployment. We hypothesize that this degradation stems from intervening on the residual stream, which indiscriminately affects aggregated features and inadvertently amplifies off-target noise. In this work, we identify a sparse subset of attention heads (only three heads) that independently govern persona and style formation, which we term Style Modulation Heads. Specifically, these heads can be localized via geometric analysis of internal representations, combining layer-wise cosine similarity and head-wise contribution scores. We demonstrate that intervention targeting only these specific heads achieves robust behavioral control while significantly mitigating the coherency degradation observed in residual stream steering. More broadly, our findings show that precise, component-level localization enables safer and more precise model control.

Related papers

• Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation [74.17379276939599]
  We show that activation steering can induce broad misalignment, even in the recent Qwen-3.5 series.<n>We characterize properties of AS-induced EM by analyzing key steering-specific factors, including steering magnitude, the low-rank structure of the steering subspace, and the number of epochs during steering-vector construction.
  arXiv  Detail & Related papers  (2026-06-07T15:34:59Z)
• Mitigating Error Amplification in Fast Adversarial Training [58.74042726356826]
  Fast Adversarial Training (FAT) has proven effective in enhancing model robustness by encouraging networks to learn perturbation-invariant representations.<n>FAT often suffers from catastrophic overfitting (CO), where the model overfits to the training attack and fails to generalize to unseen ones.<n>We propose a Distribution-aware Dynamic Guidance (DDG) strategy that dynamically adjusts both the perturbation budget and supervision signal.
  arXiv  Detail & Related papers  (2026-04-27T11:23:18Z)
• Engineering Verifiable Modularity in Transformers via Per-Layer Supervision [0.0]
  Ablating an attention head identified as critical for capitalization produces minimal behavioral change because distributed redundancy compensates for damage.<n>We demonstrate that architectural interventions can expose hidden modularity.<n>This es tablishes a methodology for transforming interpretability from passive observation to active control.
  arXiv  Detail & Related papers  (2026-03-08T05:18:14Z)
• Steering Safely or Off a Cliff? Rethinking Specificity and Robustness in Inference-Time Interventions [2.977664945581083]
  We show that while steering achieves high efficacy and largely maintains general and control specificity, it consistently fails to preserve robustness specificity.<n>Our work provides the first systematic evaluation of specificity in model steering, showing that standard efficacy and specificity checks are insufficient.
  arXiv  Detail & Related papers  (2026-02-05T23:14:05Z)
• Internalizing LLM Reasoning via Discovery and Replay of Latent Actions [4.830503861275364]
  Internalization of chain-of-thought processes into hidden states has emerged as a highly efficient paradigm for scaling test-time compute.<n>We propose STIR (Self-Distilled Tools for Internal Reasoning), a framework that reformulates reasoning enhancement as a dynamic latent trajectory control problem.
  arXiv  Detail & Related papers  (2026-02-04T08:44:57Z)
• Why Steering Works: Toward a Unified View of Language Model Parameter Dynamics [81.80010043113445]
  Local weight fine-tuning, LoRA-based adaptation, and activation-based interventions are studied in isolation.<n>We present a unified view that frames these interventions as dynamic weight updates induced by a control signal.<n>Across methods, we observe a consistent trade-off between preference and utility: stronger control increases preference while predictably reducing utility.
  arXiv  Detail & Related papers  (2026-02-02T17:04:36Z)
• BLOCK-EM: Preventing Emergent Misalignment by Blocking Causal Features [6.495737609776765]
  Emergent misalignment can arise when a language model is fine-tuned on a narrowly scoped supervised objective.<n>We investigate a mechanistic approach to preventing emergent misalignment by identifying a small set of internal features that reliably control the misaligned behavior.
  arXiv  Detail & Related papers  (2026-01-31T15:11:05Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Models [48.385105851598745]
  "HeadHunter" is a systematic framework for iteratively selecting attention heads that align with user-centric objectives.<n>"SoftPAG" is a continuous knob to tune perturbation strength and suppress artifacts.<n>We validate our method on modern large-scale DiT-based text-to-image models.
  arXiv  Detail & Related papers  (2025-06-12T17:59:51Z)
• Semi-Supervised Unconstrained Head Pose Estimation in the Wild [57.11798881492183]
  We propose the first semi-supervised unconstrained head pose estimation method SemiUHPE.<n>Our method is based on the observation that the aspect-ratio invariant cropping of wild heads is superior to previous landmark-based affine alignment.<n>Our proposed method is also beneficial for solving other closely related problems, including generic object rotation regression and 3D head reconstruction.
  arXiv  Detail & Related papers  (2024-04-03T08:01:00Z)
• Learning Action-based Representations Using Invariance [18.1941237781348]
  We introduce action-bisimulation encoding, which learns a multi-step controllability metric that discounts distant state features that are relevant for control. We demonstrate that action-bisimulation pretraining on reward-free, uniformly random data improves sample efficiency in several environments.
  arXiv  Detail & Related papers  (2024-03-25T02:17:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.