Non-linear Interventions on Large Language Models

• URL: http://arxiv.org/abs/2605.14749v1
• Date: Thu, 14 May 2026 12:14:42 GMT
• Title: Non-linear Interventions on Large Language Models
• Abstract summary: We introduce a general formulation of intervention that extends naturally to non-linearly represented features.<n>We validate our framework on refusal bypass steering, where it steers the model more precisely than linear baselines by intervening on a non-linear feature governing refusal.
• Score: 9.941320667191393
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Intervention is one of the most representative and widely used methods for understanding the internal representations of large language models (LLMs). However, existing intervention methods are confined to linear interventions grounded in the Linear Representation Hypothesis, leaving features encoded along non-linear manifolds beyond their reach. In this work, we introduce a general formulation of intervention that extends naturally to non-linearly represented features, together with a learning procedure that further enables intervention on implicit features lacking a direct output signature. We validate our framework on refusal bypass steering, where it steers the model more precisely than linear baselines by intervening on a non-linear feature governing refusal.

Related papers

• Non-Parametric Rehearsal Learning via Conditional Mean Embeddings [88.89267783967263]
  We propose a non-parametric rehearsal learning approach for textitavoiding undesired future (AUF) problems.<n>Specifically, we use kernel machinery to reformulate the AUF objective into a unified representation that disentangles desirability modeling from action-induced distributional changes.
  arXiv  Detail & Related papers  (2026-05-09T15:30:52Z)
• Curveball Steering: The Right Direction To Steer Isn't Always Linear [7.264147948470562]
  Activation steering is a widely used approach for controlling large language model (LLM) behavior by intervening on internal representations.<n>We propose "Curveball steering", a nonlinear steering method based on PCA kernel that performs interventions in a feature space.
  arXiv  Detail & Related papers  (2026-03-10T07:45:35Z)
• Patches of Nonlinearity: Instruction Vectors in Large Language Models [47.132696509843164]
  We investigate how instruction-specific representations are constructed and utilized in different stages of post-training.<n>We find that instruction representation is fairly localized in models.<n>To disentangle the non-linear causal interaction, we propose a novel method to localize information processing in language models.
  arXiv  Detail & Related papers  (2026-02-08T11:56:39Z)
• Importance Sampling for Nonlinear Models [5.421981644827842]
  We introduce the concept of the adjoint operator of a nonlinear map.<n>We demonstrate that sampling based on these notions of norm and leverage scores provides approximation guarantees for the underlying nonlinear mapping.
  arXiv  Detail & Related papers  (2025-05-18T10:34:39Z)
• Minimizing False-Positive Attributions in Explanations of Non-Linear Models [5.186535458271726]
  Suppressor variables can influence model predictions without being dependent on the target outcome.<n>These variables may cause false-positive feature attributions, undermining the utility of explanations.<n>We introduce PatternLocal, a novel XAI technique that addresses this gap.
  arXiv  Detail & Related papers  (2025-05-16T13:06:12Z)
• Adversarial Dependence Minimization [78.36795688238155]
  This work provides a differentiable and scalable algorithm for dependence minimization that goes beyond linear pairwise decorrelation.<n>We demonstrate its utility in three applications: extending PCA to nonlinear decorrelation, improving the generalization of image classification methods, and preventing dimensional collapse in self-supervised representation learning.
  arXiv  Detail & Related papers  (2025-02-05T14:43:40Z)
• Representation Surgery: Theory and Practice of Affine Steering [72.61363182652853]
  Language models often exhibit undesirable behavior, e.g., generating toxic or gender-biased text.<n>One natural (and common) approach to prevent the model from exhibiting undesirable behavior is to steer the model's representations.<n>This paper investigates the formal and empirical properties of steering functions.
  arXiv  Detail & Related papers  (2024-02-15T00:20:30Z)
• Learning Linear Causal Representations from Interventions under General Nonlinear Mixing [52.66151568785088]
  We prove strong identifiability results given unknown single-node interventions without access to the intervention targets. This is the first instance of causal identifiability from non-paired interventions for deep neural network embeddings.
  arXiv  Detail & Related papers  (2023-06-04T02:32:12Z)
• Log-linear Guardedness and its Implications [116.87322784046926]
  Methods for erasing human-interpretable concepts from neural representations that assume linearity have been found to be tractable and useful. This work formally defines the notion of log-linear guardedness as the inability of an adversary to predict the concept directly from the representation. We show that, in the binary case, under certain assumptions, a downstream log-linear model cannot recover the erased concept.
  arXiv  Detail & Related papers  (2022-10-18T17:30:02Z)
• Linear Adversarial Concept Erasure [108.37226654006153]
  We formulate the problem of identifying and erasing a linear subspace that corresponds to a given concept.<n>We show that the method is highly expressive, effectively mitigating bias in deep nonlinear classifiers while maintaining tractability and interpretability.
  arXiv  Detail & Related papers  (2022-01-28T13:00:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.