Steering Code LLMs with Activation Directions for Language and Library Control

• URL: http://arxiv.org/abs/2603.23629v2
• Date: Thu, 26 Mar 2026 20:57:47 GMT
• Title: Steering Code LLMs with Activation Directions for Language and Library Control
• Abstract summary: Code LLMs often default to particular programming languages and libraries under neutral prompts.<n>We investigate whether these preferences are encoded as approximately linear directions in activation space that can be manipulated at inference time.
• Score: 10.49462610276906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code LLMs often default to particular programming languages and libraries under neutral prompts. We investigate whether these preferences are encoded as approximately linear directions in activation space that can be manipulated at inference time. Using a difference-in-means method, we estimate layer-wise steering vectors for five language/library pairs and add them to model hidden states during generation. Across three open-weight code LLMs, these interventions substantially increase generation toward the target ecosystem under neutral prompts and often remain effective even when prompts explicitly request the opposite choice. Steering strength varies by model and target, with common ecosystems easier to induce than rarer alternatives, and overly strong interventions can reduce output quality. Overall, our results suggest that code-style preferences in LLMs are partly represented by compact, steerable structure in activation space.

Related papers

• UniSteer: Text-Guided Flow Matching in Activation Space for Versatile LLM Steering [42.82337069202406]
  Activation-based control has emerged as an effective paradigm for controlling behaviors such as persona and style.<n>We propose UniSteer, a text-guided activation flow matching model that learns a conditional distribution over residual-stream activations.<n>We show that UniSteer provides a unified interface across behavioral control, truthfulness steering, fine-grained concept steering, multi-constraint instruction following, and activation-space classification.
  arXiv  Detail & Related papers  (2026-05-28T15:23:29Z)
• Neuro-Symbolic Control with Large Language Models for Language-Guided Spatial Tasks [0.769971486557519]
  A neuro-symbolic control framework is proposed that distinguishes between low-level motion execution and high-level semantic reasoning.<n>We show that the framework enhances interpretability, stability, and generalization without any need of reinforcement learning or costly rollouts.
  arXiv  Detail & Related papers  (2025-12-19T08:08:40Z)
• Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space [57.868527884634894]
  Natural Language Actor-Critic is a novel actor-critic algorithm that trains policies using natural language rather than scalar values.<n>We present results on a mixture of reasoning, web browsing, and tool-use with dialogue tasks, demonstrating that NLAC shows promise in outperforming existing training approaches.
  arXiv  Detail & Related papers  (2025-12-04T09:21:44Z)
• Steering Conceptual Bias via Transformer Latent-Subspace Activation [0.0]
  This work examines whether activating latent subspaces in language models (LLMs) can steer scientific code generation toward a specific programming language.<n>A neuron-attribution method, perturbing the highest activated static weight for a C++ or CPP token, proved brittle and exhibited limited generalization.<n>A gradient-refined adaptive activation steering framework (G-ACT) was developed.
  arXiv  Detail & Related papers  (2025-06-23T17:56:34Z)
• Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMs [8.085475675888045]
  Activation steering provides an alternative for inference-time control.<n>We introduce a novel approach using a lightweight, trainable controller network integrated during inference.
  arXiv  Detail & Related papers  (2025-05-22T01:48:38Z)
• Navigating Motion Agents in Dynamic and Cluttered Environments through LLM Reasoning [69.5875073447454]
  This paper advances motion agents empowered by large language models (LLMs) toward autonomous navigation in dynamic and cluttered environments.<n>Our training-free framework supports multi-agent coordination, closed-loop replanning, and dynamic obstacle avoidance without retraining or fine-tuning.
  arXiv  Detail & Related papers  (2025-03-10T13:39:09Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language Models [16.37602070339033]
  Large Language Models (LLMs) often generate inconsistent responses when prompted with semantically equivalent paraphrased inputs.<n>We propose LF-Steering, a novel activation steering approach to precisely identify latent feature representations responsible for semantic inconsistency.<n>Our method maps the hidden states of the relevant transformer layer into a sparsely activated, high-dimensional feature space based on a sparse autoencoder.
  arXiv  Detail & Related papers  (2025-01-19T13:06:51Z)
• From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control [58.72492647570062]
  We introduce our method -- Learnable Latent Codes as Bridges (LCB) -- as an alternate architecture to overcome limitations.<n>We find that methodoutperforms baselines that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.
  arXiv  Detail & Related papers  (2024-05-08T04:14:06Z)
• DECIDER: A Dual-System Rule-Controllable Decoding Framework for Language Generation [57.07295906718989]
  Constrained decoding approaches aim to control the meaning or style of text generated by pre-trained large language (Ms also PLMs) for various tasks at inference time.<n>These methods often guide plausible continuations by greedily and explicitly selecting targets.<n>Inspired by cognitive dual-process theory, we propose a novel decoding framework DECIDER.
  arXiv  Detail & Related papers  (2024-03-04T11:49:08Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Let Models Speak Ciphers: Multiagent Debate through Embeddings [84.20336971784495]
  We introduce CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue. By deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights. This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.
  arXiv  Detail & Related papers  (2023-10-10T03:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.