In-Context Learning Operates as Concept Subspace Learning

• URL: http://arxiv.org/abs/2605.18830v1
• Date: Tue, 12 May 2026 21:43:48 GMT
• Title: In-Context Learning Operates as Concept Subspace Learning
• Abstract summary: We study whether structured demonstrations induce low-dimensional concept inference.<n>For ridge and least-squares ICL proxies, prediction decomposes exactly into concept-coordinate regression and off-subspace leakage.<n>These results support concept subspaces as compact, task-aligned mediators of recoverable ICL behavior.
• Score: 20.051573945788963
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Regression and Bayesian accounts of in-context learning (ICL) explain how demonstrations can induce predictors, while mechanistic analyses often identify compact activation directions that steer prompted behavior. However, it remains unclear whether structured demonstrations induce low-dimensional concept inference. We study this question through a concept-subspace view of ICL, in which tasks vary only along intrinsic concept coordinates, although inputs are observed in a high-dimensional ambient space. For ridge and least-squares ICL proxies, prediction decomposes exactly into concept-coordinate regression and off-subspace leakage. Under block-diagonal or near-block-diagonal covariance assumptions, the leading estimation and nuisance-sensitivity terms scale with the dimension of the concept subspace, while residual effects are controlled by cross-subspace coupling. This separation gives a mechanistic prediction: recoverable task information should concentrate in a low-dimensional, task-aligned activation subspace. On CounterFact-derived multi-relation prompts with Llama-3-8B, a 68--73-dimensional subspace of the 4096-dimensional residual stream restores 78.8% of the clean--corrupted accuracy gap, whereas patching the complementary subspace restores 0%. Concept swaps redirect predictions toward injected relations, while random and cross-task matched-rank controls are largely ineffective. Additional experiments on Qwen2.5-7B and a controlled cross-lingual rule task show the same qualitative pattern. These results support concept subspaces as compact, task-aligned mediators of recoverable ICL behavior in structured task families, without implying full-circuit recovery.

Related papers

• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• Stochastic Linear Bandits with Partially Observed Actions [11.633556389435244]
  We study a partially observed variant of the linear bandit problem in which the learning agent only sees a random subset of coordinates for each action.<n>We show that this barrier can be overcome when the action vectors have low intrinsic dimension.<n>Our theory shows that TOFU-POV enjoys a $sqrtT$ regret that scales with the intrinsic action subspace dimension.
  arXiv  Detail & Related papers  (2026-07-09T22:23:17Z)
• Subspace-Decomposed JEPAs: Disentangling Progression and Content in Latent World Models [1.034052616244602]
  Joint-Embedding Predictive Architectures learn compact latent world models by predicting future embeddings.<n>We carve the JEPA latent into two subspaces with disjoint roles.<n>We prove that the two anti-collapse forces act on disjoint coordinates, so they compose additively rather than competing on the same dimensions.
  arXiv  Detail & Related papers  (2026-05-29T10:23:48Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing [3.4002388038666105]
  Reservoir computing typically relies on large, randomly generated reservoirs, enabling simple, often linear readouts.<n>We develop reservoir design principles from a geometric perspective for inputs generated by deterministic dynamical systems.<n> Numerical experiments demonstrate consistent performance gains over arbitrary reservoir constructions.
  arXiv  Detail & Related papers  (2026-05-24T19:13:13Z)
• Beyond Activation Alignment: The Geometry of Neural Sensitivity [0.0]
  We introduce a framework based on local decodable information, which focuses on a representation's ability to discriminate small perturbations.<n>We compare these regularized signatures using a log-spectral distance on the manifold of symmetric positive definite (SPD) matrices.<n> Empirically, this framework enables the recovery of corresponding layers across independently trained artificial neural networks.
  arXiv  Detail & Related papers  (2026-05-04T23:21:24Z)
• Rethinking Transferable Adversarial Attacks on Point Clouds from a Compact Subspace Perspective [55.919842734983156]
  CoSA is a transferable attack framework that operates within a shared low-dimensional semantic space.<n>CoSA consistently outperforms state-of-the-art transferable attacks.
  arXiv  Detail & Related papers  (2026-01-30T15:48:11Z)
• Sparse Concept Anchoring for Interpretable and Controllable Neural Representations [0.9831489366502301]
  We introduce Sparse Concept Anchoring, a method that biases latent space to position a targeted subset of concepts.<n>The anchored geometry enables two practical interventions: behavioral steering that projects out a concept's latent component at inference, and permanent removal.
  arXiv  Detail & Related papers  (2025-12-13T21:43:17Z)
• Q-function Decomposition with Intervention Semantics with Factored Action Spaces [51.01244229483353]
  We consider Q-functions defined over a lower dimensional projected subspace of the original action space, and study the condition for the unbiasedness of decomposed Q-functions.<n>This leads to a general scheme which we call action decomposed reinforcement learning that uses the projected Q-functions to approximate the Q-function in standard model-free reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2025-04-30T05:26:51Z)
• Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals [52.123343364599094]
  adversarial attacks place carefully crafted perturbations on normal examples to fool deep neural networks (DNNs) We first empirically show that the features of either clean signals or adversarial perturbations are redundant and span in low-dimensional linear subspaces respectively with minimal overlap. This makes it possible for DNNs to learn a subspace where only features of clean signals exist while those of perturbations are discarded.
  arXiv  Detail & Related papers  (2024-03-24T14:35:44Z)
• A Geometric Notion of Causal Probing [85.49839090913515]
  The linear subspace hypothesis states that, in a language model's representation space, all information about a concept such as verbal number is encoded in a linear subspace.<n>We give a set of intrinsic criteria which characterize an ideal linear concept subspace.<n>We find that, for at least one concept across two languages models, the concept subspace can be used to manipulate the concept value of the generated word with precision.
  arXiv  Detail & Related papers  (2023-07-27T17:57:57Z)
• SubspaceNet: Deep Learning-Aided Subspace Methods for DoA Estimation [36.647703652676626]
  SubspaceNet is a data-driven DoA estimator which learns how to divide the observations into distinguishable subspaces. SubspaceNet is shown to enable various DoA estimation algorithms to cope with coherent sources, wideband signals, low SNR, array mismatches, and limited snapshots.
  arXiv  Detail & Related papers  (2023-06-04T06:30:13Z)
• Unsupervised Discriminative Embedding for Sub-Action Learning in Complex Activities [54.615003524001686]
  This paper proposes a novel approach for unsupervised sub-action learning in complex activities. The proposed method maps both visual and temporal representations to a latent space where the sub-actions are learnt discriminatively. We show that the proposed combination of visual-temporal embedding and discriminative latent concepts allow to learn robust action representations in an unsupervised setting.
  arXiv  Detail & Related papers  (2021-04-30T20:07:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.