ORBIT: Training-Free Multi-Attribute Behavioral Steering via Orthogonal Subspace Rotation

• URL: http://arxiv.org/abs/2606.22357v1
• Date: Sun, 21 Jun 2026 06:40:32 GMT
• Title: ORBIT: Training-Free Multi-Attribute Behavioral Steering via Orthogonal Subspace Rotation
• Abstract summary: We introduce ORBIT, a training-free extension of rotation-based steering to the multi-attribute setting.<n>Our method constructs a joint subspace from per-attribute steering planes via singular value decomposition.<n>We also introduce TraitFactory, a new multi-attribute benchmark that focuses on behavioral tendencies rather than surface-level style.
• Score: 38.36944876774961
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models are widely used in assistant settings, where controlling behavioral attributes is often essential. Activation steering modifies hidden-state representations at inference time, providing a lightweight, training-free mechanism that can be toggled at runtime. Existing methods, however, have focused primarily on steering a single attribute at a time. When multiple attributes must be controlled simultaneously, naive summation of per-attribute steering vectors suffers from norm imbalance and directional cancellation, while classifier-based approaches require retraining whenever the attribute set changes. We introduce ORBIT (Orthogonal Rotation-Based Intervention Technique), a training-free extension of rotation-based steering to the multi-attribute setting. Our method constructs a joint subspace from per-attribute steering planes via singular value decomposition and applies a single norm-preserving rotation within that subspace toward a combined target direction. Adaptive per-token gating identifies which attributes need correction at each position, and an optional additive boost strengthens attributes with weak initial projection. We also introduce TraitFactory, a new multi-attribute benchmark that focuses on behavioral tendencies rather than surface-level style. We evaluate ORBIT on TraitFactory and ToneBank across three models (Llama-3.2-3B, Qwen-2.5-7B, Llama-3.1-8B) while steering multiple attributes simultaneously, showing that it achieves stronger and more balanced multi-attribute steering than existing training-free baselines while better preserving output coherence.

Related papers

• DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection [16.74080857973598]
  We propose a fully automated anomaly detection framework leveraging global context to remove manual tunings.<n>Our method achieves Image-AUC scores of 97.7%, 93.2%, and 84.5% on MVTec-AD, VisA, and Real-IAD datasets.
  arXiv  Detail & Related papers  (2026-07-27T01:31:48Z)
• Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior [13.073472989807675]
  INNSteer is a nonlinear activation steering framework based on invertible latent transformations.<n>It consistently improves model control over linear, transport-based, and nonlinear steering baselines.
  arXiv  Detail & Related papers  (2026-06-07T05:01:25Z)
• Selective Steering: Norm-Preserving Control Through Discriminative Layer Selection [1.7802147489386628]
  Large language models (LLMs) remain vulnerable to adversarial attacks that elicit harmful behaviors.<n>We propose Selective Steering, which addresses these limitations through two key innovations.<n> Experiments across nine models demonstrate that Selective Steering achieves 5.5x higher attack success rates than prior methods.
  arXiv  Detail & Related papers  (2026-01-27T08:56:25Z)
• RISER: Orchestrating Latent Reasoning Skills for Adaptive Activation Steering [62.63376387138257]
  We propose a plug-and-play intervention framework that adaptively steers large language models (LLMs) reasoning in activation space.<n>RISER constructs a library of reusable reasoning vectors and employs a lightweight Router to dynamically compose them for each input.<n>The Router is optimized via reinforcement learning under task-level rewards, activating latent cognitive primitives in an emergent and compositional manner.
  arXiv  Detail & Related papers  (2026-01-14T08:04:33Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)
• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)
• Beyond Linear Steering: Unified Multi-Attribute Control for Language Models [8.223479896191145]
  We introduce K-Steering, a unified and flexible approach that trains a single non-linear multi-label classifier on hidden activations.<n>This avoids linearity assumptions, removes the need for storing and tuning separate vectors attribute, and allows dynamic composition of behaviors without retraining.<n> Empirical results across 3 model families, validated by both activation-based classifiers and LLM-based judges, demonstrate that K-Steering outperforms strong baselines in accurately steering multiple behaviors.
  arXiv  Detail & Related papers  (2025-05-30T12:41:19Z)
• Multi-Attribute Steering of Language Models via Targeted Intervention [56.93583799109029]
  Inference-time intervention (ITI) has emerged as a promising method for steering large language model (LLM) behavior in a particular direction.<n>We introduce Multi-Attribute Targeted Steering (MAT-Steer), a novel steering framework designed for selective token-level intervention across multiple attributes.
  arXiv  Detail & Related papers  (2025-02-18T02:27:23Z)
• I^3Net: Implicit Instance-Invariant Network for Adapting One-Stage Object Detectors [64.93963042395976]
  Implicit Instance-Invariant Network (I3Net) is tailored for adapting one-stage detectors. I3Net implicitly learns instance-invariant features via exploiting the natural characteristics of deep features in different layers. Experiments reveal that I3Net exceeds the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2021-03-25T11:14:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.