Layerwise Dynamics for In-Context Classification in Transformers

• URL: http://arxiv.org/abs/2604.11613v2
• Date: Thu, 16 Apr 2026 18:05:44 GMT
• Title: Layerwise Dynamics for In-Context Classification in Transformers
• Abstract summary: We study multi-class linear classification in the hard no-margin regime.<n>We make computation identifiable by enforcing feature- and label-permutation equivariance at every layer.<n>This enables interpretability while maintaining functional equivalence.
• Score: 27.052342498221748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers can perform in-context classification from a few labeled examples, yet the inference-time algorithm remains opaque. We study multi-class linear classification in the hard no-margin regime and make the computation identifiable by enforcing feature- and label-permutation equivariance at every layer. This enables interpretability while maintaining functional equivalence and yields highly structured weights. From these models we extract an explicit depth-indexed recursion: an end-to-end identified, emergent update rule inside a softmax transformer, to our knowledge the first of its kind. Attention matrices formed from mixed feature-label Gram structure drive coupled updates of training points, labels, and the test probe. The resulting dynamics implement a geometry-driven algorithmic motif, which can provably amplify class separation and yields robust expected class alignment.

Related papers

• Complexity Induction: Compositional Generalization via Structured Training Distortion [51.56484100374058]
  We show that structured distortion of training data can induce compositional generalization without architectural modification.<n>Results suggest that structured complication of training signals can influence both the internal organization of learned representations and their compositional interpretation.
  arXiv  Detail & Related papers  (2026-08-21T00:45:02Z)
• Tensor Network Machine Learning for Wildfire Susceptibility Mapping: from Grokking Dynamics to Quantum Mixedness of Class Representations [53.66343110413339]
  A quantum-inspired tensor network framework for wildfire susceptibility classification in the Gargano region is introduced.<n>The approach combines scalable geospatial representations with an interpretable quantum mask, enabling both binary and multiclass classification.
  arXiv  Detail & Related papers  (2026-07-21T18:30:50Z)
• Self-supervised Latent Space Optimization with Nebula Variational Coding [87.20343320266215]
  This paper proposes a variational inference model which leads to a clustered embedding.<n>We introduce additional variables in the latent space, called textbfnebula anchors, that guide the latent variables to form clusters during training.<n>Since each latent feature can be labeled with the closest anchor, we also propose to apply metric learning in a self-supervised way to make the separation between clusters more explicit.
  arXiv  Detail & Related papers  (2025-06-02T08:13:32Z)
• Benign Overfitting and the Geometry of the Ridge Regression Solution in Binary Classification [75.01389991485098]
  We show that ridge regression has qualitatively different behavior depending on the scale of the cluster mean vector.<n>In regimes where the scale is very large, the conditions that allow for benign overfitting turn out to be the same as those for the regression task.
  arXiv  Detail & Related papers  (2025-03-11T01:45:42Z)
• Distinguishability Calibration to In-Context Learning [31.375797763897104]
  We propose a method to map a PLM-encoded embedding into a new metric space to guarantee the distinguishability of the resulting embeddings. We also take the advantage of hyperbolic embeddings to capture the hierarchical relations among fine-grained class-associated token embedding.
  arXiv  Detail & Related papers  (2023-02-13T09:15:00Z)
• Exploring Category-correlated Feature for Few-shot Image Classification [27.13708881431794]
  We present a simple yet effective feature rectification method by exploring the category correlation between novel and base classes as the prior knowledge. The proposed approach consistently obtains considerable performance gains on three widely used benchmarks.
  arXiv  Detail & Related papers  (2021-12-14T08:25:24Z)
• On the rate of convergence of a classifier based on a Transformer encoder [55.41148606254641]
  The rate of convergence of the misclassification probability of the classifier towards the optimal misclassification probability is analyzed. It is shown that this classifier is able to circumvent the curse of dimensionality provided the aposteriori probability satisfies a suitable hierarchical composition model.
  arXiv  Detail & Related papers  (2021-11-29T14:58:29Z)
• Learning from Multiple Annotators by Incorporating Instance Features [15.643325526074804]
  Learning from multiple annotators aims to induce a high-quality classifier from training instances. Most existing methods adopt class-level confusion matrices of annotators that observed labels do not depend on the instance features. We propose a noise transition matrix, which incorporates the influence of instance features on annotators' performance based on confusion matrices.
  arXiv  Detail & Related papers  (2021-06-29T08:07:24Z)
• Convolutional Dynamic Alignment Networks for Interpretable Classifications [108.83345790813445]
  We introduce a new family of neural network models called Convolutional Dynamic Alignment Networks (CoDA-Nets) Their core building blocks are Dynamic Alignment Units (DAUs), which linearly transform their input with weight vectors that dynamically align with task-relevant patterns. CoDA-Nets model the classification prediction through a series of input-dependent linear transformations, allowing for linear decomposition of the output into individual input contributions.
  arXiv  Detail & Related papers  (2021-03-31T18:03:53Z)
• Provably End-to-end Label-Noise Learning without Anchor Points [118.97592870124937]
  We propose an end-to-end framework for solving label-noise learning without anchor points. Our proposed framework can identify the transition matrix if the clean class-posterior probabilities are sufficiently scattered.
  arXiv  Detail & Related papers  (2021-02-04T03:59:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.