When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search

• URL: http://arxiv.org/abs/2605.16362v1
• Date: Sat, 09 May 2026 14:26:49 GMT
• Title: When Is Rank-1 Steering Cheap? Geometry, Granularity, and Budgeted Search
• Abstract summary: Activation steering offers a lightweight way to control LLMs without retraining, but its effectiveness varies sharply across concepts.<n>We formalize rank-1 steering as a budget-constrained optimization over intervention layer and coefficient.<n>We present textitGRACE, a Granularity- and Representation-Aware Concept Engineering framework that uses activation geometry to diagnose the dominant source of steering difficulty.
• Score: 61.73232505100331
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Activation steering offers a lightweight way to control LLMs without retraining, but its effectiveness varies sharply across concepts. Prior work often reads this variability as evidence that many concepts are not captured by a single steering direction. We argue instead that much of it reflects search difficulty: a useful rank-1 intervention often exists, but finding it can be expensive. We formalize rank-1 steering as a budget-constrained optimization over intervention layer and coefficient. Across concepts and model families, prompt-boundary directional alignment predicts where effective interventions occur, enabling geometry-guided search that reaches high utility with substantially fewer evaluations, reducing the trials needed to recover 95\% of best-found utility by 39.8\% on average across three model families. To explain why some concepts remain expensive even under better search, we introduce \emph{concept granularity}, a measure of directional heterogeneity across contrastive contexts. Granularity distinguishes concepts whose difference vectors share a stable global direction from those where prompts agree locally within each input but the utility-maximizing direction rotates systematically across inputs. Higher granularity is associated with slower convergence and lower best-found performance (Pearson $r{=}0.44$ with trials-to-95\%, $r{=}{-}0.46$ with best-found utility, both $p<0.001$). We present \textit{GRACE}, a Granularity- and Representation-Aware Concept Engineering framework that uses activation geometry to diagnose the dominant source of steering difficulty, select the appropriate remedy, and allocate optimization effort efficiently. Our results shift the frame from ``\textit{when does rank-1 fail?}'' to ``\textit{when is rank-1 cheap and stable?}'', turning activation geometry from a descriptive tool into an actionable prior for LLM control.

Related papers

• Adaptively Incorporating Directional Hints into Zeroth-Order Optimization [49.940317286021696]
  We develop an algorithm that achieves zeroth-order optimization of non- functions with the aid directional hints.<n>We validate the method, demonstrating sustained progress on non- control tasks.
  arXiv  Detail & Related papers  (2026-09-08T05:42:02Z)
• Steering Geometry: Validating Human Value Geometry in LLM Steering Space [14.117484627215804]
  Large language models (LLMs) are increasingly deployed in alignment-sensitive contexts.<n>We investigate whether the latent geometry of LLM steering vectors reflects theory-specified structure in human values and morality.<n>We find that distribution-driven methods recover human value topologies aligned with theoretical predictions.
  arXiv  Detail & Related papers  (2026-09-05T22:55:48Z)
• One Policy, Any Budget: Internalizing Budget-Aware Search via Reinforcement Learning [41.55260785497756]
  We propose AnySearch, a framework that enables a single policy to perform budget-aware search under any budget constraint.<n>In the first phase, we train the agent with explicit budget state injection and structured reasoning prompts.<n>In the second phase, the scaffold is removed and the agent learns to operate autonomously under adaptively sampled budget constraints.
  arXiv  Detail & Related papers  (2026-09-01T07:12:32Z)
• Robust Activation Map Rectification for Weakly Supervised Volumetric Segmentation: Temporal Coherence as a Free Lunch [31.156115062163412]
  Weakly supervised segmentation relies heavily on class activation maps (CAMs) to initially localize target regions.<n>Existing remedies typically introduce additional training stages or prototype learning, increasing computational cost and reducing robustness.<n>We propose a training-free prototype-free framework that rectifies unreliable CAMs by exploiting temporal and structural coherence in volumetric data as a free lunch.
  arXiv  Detail & Related papers  (2026-07-22T08:07:38Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation [0.0]
  Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models.<n>We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions.
  arXiv  Detail & Related papers  (2026-02-10T22:45:47Z)
• When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning [18.50038551163168]
  We show that low-rank settings, especially rank-1 LoRA, are often unstable.<n>We propose Gap-Init to align rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set.<n>Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.
  arXiv  Detail & Related papers  (2026-02-02T01:31:25Z)
• Directional Reasoning Injection for Fine-Tuning MLLMs [51.53222423215055]
  Multimodal large language models (MLLMs) are rapidly advancing, yet their reasoning ability often lags behind that of strong text-only counterparts.<n>Existing methods to bridge this gap rely on supervised fine-tuning over large-scale multimodal reasoning data or reinforcement learning.<n>We propose Directional Reasoning Injection for Fine-Tuning (DRIFT) to solve this problem.
  arXiv  Detail & Related papers  (2025-10-16T18:06:46Z)
• Amortized Latent Steering: Low-Cost Alternative to Test-Time Optimization [3.9311957222075935]
  Amortized Latent Steering (ALS) collapses iterative optimization into a single offline-computed vector.<n>ALS achieves $2-5times$ speedup over iterative methods while matching or surpassing greedy Chain-of-Thought (CoT) and Self-Consistency baselines.<n>Results show that much of latent optimization's benefit can be captured offline, making sophisticated reasoning techniques viable for production deployment.
  arXiv  Detail & Related papers  (2025-09-10T07:03:35Z)
• AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders [73.37603699731329]
  We introduce AxBench, a large-scale benchmark for steering and concept detection.<n>For steering, we find that prompting outperforms all existing methods, followed by finetuning.<n>For concept detection, representation-based methods such as difference-in-means, perform the best.
  arXiv  Detail & Related papers  (2025-01-28T18:51:24Z)
• Cost Aware Best Arm Identification [13.380383930882784]
  We call it emphCost Aware Best Arm Identification (CABAI) We propose a simple algorithm called emphChernoff Overlap (CO), based on a square-root rule. Our results show that (i.e. ignoring the heterogeneous action cost results in sub-optimality in practice, and (ii.) simple algorithms can deliver near-optimal performance over a wide range of problems.
  arXiv  Detail & Related papers  (2024-02-26T16:27:08Z)
• Cooperative Thresholded Lasso for Sparse Linear Bandit [6.52540785559241]
  We present a novel approach to address the multi-agent sparse contextual linear bandit problem. It is first algorithm that tackles row-wise distributed data in sparse linear bandits. It is widely applicable to high-dimensional multi-agent problems where efficient feature extraction is critical for minimizing regret.
  arXiv  Detail & Related papers  (2023-05-30T16:05:44Z)
• TransPath: Learning Heuristics For Grid-Based Pathfinding via Transformers [64.88759709443819]
  We suggest learning the instance-dependent proxies that are supposed to notably increase the efficiency of the search. The first proxy we suggest to learn is the correction factor, i.e. the ratio between the instance independent cost-to-go estimate and the perfect one. The second proxy is the path probability, which indicates how likely the grid cell is lying on the shortest path.
  arXiv  Detail & Related papers  (2022-12-22T14:26:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.