Concept Heterogeneity-aware Representation Steering

• URL: http://arxiv.org/abs/2603.02237v1
• Date: Fri, 13 Feb 2026 11:07:23 GMT
• Title: Concept Heterogeneity-aware Representation Steering
• Abstract summary: Representation steering offers a mechanism for controlling the behavior of large language models (LLMs) by intervening on internal activations at inference time.<n>Most existing methods rely on a single global steering direction, typically obtained via difference-in-means over contrastive datasets.<n>In this work, we view representation steering through the lens of optimal transport (OT), noting that standard difference-in-means steering implicitly corresponds to the OT map between two unimodal Gaussian distributions.
• Score: 6.247724933679344
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representation steering offers a lightweight mechanism for controlling the behavior of large language models (LLMs) by intervening on internal activations at inference time. Most existing methods rely on a single global steering direction, typically obtained via difference-in-means over contrastive datasets. This approach implicitly assumes that the target concept is homogeneously represented across the embedding space. In practice, however, LLM representations can be highly non-homogeneous, exhibiting clustered, context-dependent structure, which renders global steering directions brittle. In this work, we view representation steering through the lens of optimal transport (OT), noting that standard difference-in-means steering implicitly corresponds to the OT map between two unimodal Gaussian distributions with identical covariance, yielding a global translation. To relax this restrictive assumption, we theoretically model source and target representations as Gaussian mixture models and formulate steering as a discrete OT problem between semantic latent clusters. From the resulting transport plan, we derive an explicit, input-dependent steering map via barycentric projection, producing a smooth, kernel-weighted combination of cluster-level shifts. We term this method Concept Heterogeneity-aware Representation Steering (CHaRS). Through numerous experimental settings, we show that CHaRS yields more effective behavioral control than global steering.

Related papers

• Disentangling Steering Vectors [13.671243729016568]
  We propose a framework to explicitly isolate individual and semantically consistent features from composite directions.<n>Specifically, we pair positive and negative activations and take their differences to generate a set of instance-level steering vectors.<n>We show that our method yields a set of semantically consistent basis vectors whose steering effects are mutually distinguishable.
  arXiv  Detail & Related papers  (2026-09-07T04:45:32Z)
• Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering [57.38215918201251]
  We propose HERL, a Hyperbolic Enhanced Representation Learning framework for Incomplete Multi-View Clustering.<n> operating within the Poincaré ball, HERL constructs a structure-aware latent space to enhance representation learning.<n>We show that HERL consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-04-18T10:50:46Z)
• Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning [37.39836347901149]
  We assess whether Large Language Models (LLMs) can generate outputs that adhere to a desired target distribution.<n>We formulate distribution alignment using the attributes of gender, race, and sentiment within occupational contexts.<n>We propose a novel fine-tuning framework that couples Steering Token with Semantic Alignment.
  arXiv  Detail & Related papers  (2026-04-07T11:58:16Z)
• Global Evolutionary Steering: Refining Activation Steering Control via Cross-Layer Consistency [12.78111885039368]
  Activation engineering enables precise control over Large Language Models (LLMs) without the computational cost of fine-tuning.<n>Existing methods deriving vectors from static activation differences are susceptible to high-dimensional noise and layer-wise semantic drift.<n>We propose Global Evolutionary Refined Steering (GER-steer), a training-free framework that grounded in the geometric stability of the network's representation evolution.
  arXiv  Detail & Related papers  (2026-03-12T03:45:19Z)
• ODESteer: A Unified ODE-Based Steering Framework for LLM Alignment [49.68063561145927]
  We propose a unified ordinary differential equations (ODEs)-based theoretical framework for activation steering.<n>We introduce ODESteer, a kind of ODE-based steering guided by barrier functions.<n>Compared to state-of-the-art activation steering methods, ODESteer achieves consistent empirical improvements.
  arXiv  Detail & Related papers  (2026-02-19T17:13:44Z)
• Faithful Bi-Directional Model Steering via Distribution Matching and Distributed Interchange Interventions [37.08071497197165]
  Intervention-based model steering offers a lightweight and interpretable alternative to prompting and fine-tuning.<n>We build on the principles of distributed alignment search to propose a new steering method: Concept DAS.<n>We show that Concept DAS does not always outperform preference-optimization methods but may benefit more from increased model scale.
  arXiv  Detail & Related papers  (2026-02-05T02:51:00Z)
• UniRoute: Unified Routing Mixture-of-Experts for Modality-Adaptive Remote Sensing Change Detection [6.323154336421137]
  UniRoute is a unified framework for modality-adaptive learning.<n>We introduce an Adaptive Receptive Field Routing MoE module to disentangle local spatial details from global semantic context.<n>We also propose a Consistency-Aware Self-Distillation strategy that stabilizes unified training under data-scarce heterogeneous settings.
  arXiv  Detail & Related papers  (2026-01-21T09:21:25Z)
• Toward Stable Semi-Supervised Remote Sensing Segmentation via Co-Guidance and Co-Fusion [31.189038928192648]
  Co2S is a semi-supervised RS segmentation framework that fuses priors from vision-language models and self-supervised models.<n>An explicit-implicit semantic co-guidance mechanism is introduced that utilizes text embeddings and learnable queries.<n>Experiments on six popular datasets demonstrate the superiority of the proposed method.
  arXiv  Detail & Related papers  (2025-12-28T18:24:19Z)
• DiffusionDriveV2: Reinforcement Learning-Constrained Truncated Diffusion Modeling in End-to-End Autonomous Driving [65.7087560656003]
  Generative diffusion models for end-to-end autonomous driving often suffer from mode collapse.<n>We propose DiffusionDriveV2, which leverages reinforcement learning to constrain low-quality modes and explore for superior trajectories.<n>This significantly enhances the overall output quality while preserving the inherent multimodality of its core Gaussian Mixture Model.
  arXiv  Detail & Related papers  (2025-12-08T17:29:52Z)
• Equivariant Goal Conditioned Contrastive Reinforcement Learning [5.019456977535218]
  Contrastive Reinforcement Learning (CRL) provides a promising framework for extracting useful structured representations from unlabeled interactions.<n>We propose Equivariant CRL, which further structures the latent space using equivariant constraints.<n>Our approach consistently outperforms strong baselines across a range of simulated tasks in both state-based and image-based settings.
  arXiv  Detail & Related papers  (2025-07-22T01:13:45Z)
• Semantic-Aligned Learning with Collaborative Refinement for Unsupervised VI-ReID [82.12123628480371]
  Unsupervised person re-identification (USL-VI-ReID) seeks to match pedestrian images of the same individual across different modalities without human annotations for model learning.<n>Previous methods unify pseudo-labels of cross-modality images through label association algorithms and then design contrastive learning framework for global feature learning.<n>We propose a Semantic-Aligned Learning with Collaborative Refinement (SALCR) framework, which builds up objective for specific fine-grained patterns emphasized by each modality.
  arXiv  Detail & Related papers  (2025-04-27T13:58:12Z)
• Mitigating Both Covariate and Conditional Shift for Domain Generalization [14.91361835243516]
  Domain generalization (DG) aims to learn a model on several source domains, hoping that the model can generalize well to unseen target domains. In this paper, a novel DG method is proposed to deal with the distribution shift via Visual Alignment and Uncertainty-guided belief Ensemble (VAUE)
  arXiv  Detail & Related papers  (2022-09-17T05:13:56Z)
• HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning [74.76431541169342]
  Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. We propose a novel hierarchical semantic-visual adaptation (HSVA) framework to align semantic and visual domains. Experiments on four benchmark datasets demonstrate HSVA achieves superior performance on both conventional and generalized ZSL.
  arXiv  Detail & Related papers  (2021-09-30T14:27:50Z)
• Closed-Form Factorization of Latent Semantics in GANs [65.42778970898534]
  A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. We propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
  arXiv  Detail & Related papers  (2020-07-13T18:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.