Quantifying LLM Attention-Head Stability: Implications for Circuit Universality

• URL: http://arxiv.org/abs/2602.16740v1
• Date: Tue, 17 Feb 2026 23:30:59 GMT
• Title: Quantifying LLM Attention-Head Stability: Implications for Circuit Universality
• Abstract summary: We study stability across-refits in increasingly complex transformer language models of various sizes.<n>Middle-layer heads are the least stable yet the most representationally distinct.<n>Deeper models exhibit stronger mid-depth divergence.
• Score: 2.6132365604376404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In mechanistic interpretability, recent work scrutinizes transformer "circuits" - sparse, mono or multi layer sub computations, that may reflect human understandable functions. Yet, these network circuits are rarely acid-tested for their stability across different instances of the same deep learning architecture. Without this, it remains unclear whether reported circuits emerge universally across labs or turn out to be idiosyncratic to a particular estimation instance, potentially limiting confidence in safety-critical settings. Here, we systematically study stability across-refits in increasingly complex transformer language models of various sizes. We quantify, layer by layer, how similarly attention heads learn representations across independently initialized training runs. Our rigorous experiments show that (1) middle-layer heads are the least stable yet the most representationally distinct; (2) deeper models exhibit stronger mid-depth divergence; (3) unstable heads in deeper layers become more functionally important than their peers from the same layer; (4) applying weight decay optimization substantially improves attention-head stability across random model initializations; and (5) the residual stream is comparatively stable. Our findings establish the cross-instance robustness of circuits as an essential yet underappreciated prerequisite for scalable oversight, drawing contours around possible white-box monitorability of AI systems.

Related papers

• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• SYNAPSE: Framework for Neuron Analysis and Perturbation in Sequence Encoding [4.507191680687611]
  In sensitive domains such as healthcare or cybersecurity, ensuring transparency, trustworthiness, and robustness is essential.<n>This work proposes SYNAPSE, a systematic, training-free framework for understanding and testing the internal behavior of Transformer models.
  arXiv  Detail & Related papers  (2026-03-09T14:18:47Z)
• Representation Stability in a Minimal Continual Learning Agent [0.0]
  We study a minimal continual learning agent designed to isolate representational dynamics from architectural complexity and optimization objectives.<n>We quantify representational change using cosine similarity between successive normalized state vectors and define a stability metric over time intervals.
  arXiv  Detail & Related papers  (2026-02-23T09:59:03Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Why Some Models Resist Unlearning: A Linear Stability Perspective [7.446140380340418]
  We frame unlearning through the lens of linear coherence stability.<n>We decompose coherence along three axes: within the retain set, within the forget set, and between them.<n>To further link data properties to forgettability, we study a two layer ReLU CNN under a signal plus noise model.<n>For empirical geometry, we show that Hessian tests and CNN heatmaps align closely with the predicted boundary, mapping the stability gradient based unlearning as a function of verification, mixing, and data/model alignment.
  arXiv  Detail & Related papers  (2026-02-03T01:47:26Z)
• Stabilizing Information Flow Entropy: Regularization for Safe and Interpretable Autonomous Driving Perception [8.543667347406286]
  We reconceptualize deep neural encoders as hierarchical communication chains that compress raw sensory inputs into task-relevant latent features.<n>We propose Eloss, a novel entropy-based regularizer designed as a lightweight, plug-and-play training objective.
  arXiv  Detail & Related papers  (2025-09-18T17:01:27Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts [1.9458156037869137]
  convergent learning is the degree to which neural systems arrive at similar internal representations.<n>We present a large-scale audit of convergent learning spanning dozens of vision models and thousands of layer-pair comparisons.<n>Findings fill critical gaps in our understanding of representational convergence, with implications for neuroscience and AI.
  arXiv  Detail & Related papers  (2025-02-26T00:04:24Z)
• Is Smoothness the Key to Robustness? A Comparison of Attention and Convolution Models Using a Novel Metric [0.0]
  Existing robustness evaluation approaches often lack theoretical generality or rely heavily on empirical assessments. We propose TopoLip, a metric based on layer-wise analysis that bridges topological data analysis and Lipschitz continuity for robustness evaluation.
  arXiv  Detail & Related papers  (2024-10-23T07:44:14Z)
• Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability [59.758009422067]
  We propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models.<n>Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan.<n> Experiments show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.
  arXiv  Detail & Related papers  (2024-09-25T11:22:29Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• A Generic Shared Attention Mechanism for Various Backbone Neural Networks [53.36677373145012]
  Self-attention modules (SAMs) produce strongly correlated attention maps across different layers. Dense-and-Implicit Attention (DIA) shares SAMs across layers and employs a long short-term memory module. Our simple yet effective DIA can consistently enhance various network backbones.
  arXiv  Detail & Related papers  (2022-10-27T13:24:08Z)
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain [41.66817277929783]
  We test whether 17 unsupervised, weakly supervised, and fully supervised representation learning approaches correctly infer the generative factors of variation in simple datasets. We train and test 2000+ models and observe that all of them struggle to learn the underlying mechanism regardless of supervision signal and architectural bias.
  arXiv  Detail & Related papers  (2021-07-17T11:24:18Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.