Interior interpretability with attention rollout: contraction and propagation profiles in Transformers

• URL: http://arxiv.org/abs/2607.22367v1
• Date: Fri, 24 Jul 2026 14:51:28 GMT
• Title: Interior interpretability with attention rollout: contraction and propagation profiles in Transformers
• Abstract summary: We introduce emphinterior interpretability, a propagation-based perspective on internal model organization.<n>We interpret rollout as a row-stochastic operator encoding attention-mediated propagation between feature tokens.
• Score: 17.72586636966418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Feature-attribution methods assign scores relating input variables to a model's output, but do not by themselves characterize how explicitly defined interaction operators compose across its intermediate layers. We introduce \emph{interior interpretability}, a propagation-based perspective on internal model organization, and instantiate it for tabular Transformers using attention rollout. We interpret rollout as a row-stochastic operator encoding attention-mediated propagation between feature tokens. By applying classical Doeblin--Dobrushin contraction theory, we show that a rollout operator with a small Dobrushin coefficient is quantitatively close to a rank-one stochastic matrix whose common row is determined by its normalized column sums. This result gives a structural interpretation to the corresponding rollout propagation profile. In Transformers trained for metabolomic age prediction, the measured rollout contraction strengthens with depth. Trained and randomly initialized models also exhibit different propagation profiles, although the present experiments do not establish the predictive relevance of individual rollout-ranked variables. Exploratory comparisons with PCA and GradientExplainer approximations to SHAP reveal localized agreement among highly ranked variables but weak agreement across complete rankings. Attention rollout is therefore used here as a diagnostic of attention-mediated propagation, not as a causal explanation or faithful attribution of the complete Transformer.

Related papers

• Tracing distinguishability through transformer processing with stochastic LayerNorm [0.0]
  We give representations volume, turning similarity into statistical distinguishability.<n>We realize this idea in pretrained transformers through a light-touch modification to LayerNorm.<n>These results establish distinguishability as a functionally grounded lens on transformer computation.
  arXiv  Detail & Related papers  (2026-08-31T12:57:38Z)
• ControlledShifts: Towards Standardizing Robustness Evaluation in Trajectory Prediction Under Distribution Shifts [12.370065954075299]
  Trajectory prediction is central to safety in autonomous driving.<n>Many methods attempt to mitigate distribution shift degradation through data-centric or test-time adaptation approaches.<n>We introduce ControlledShifts, a framework that systematically re-splits existing trajectory datasets into in-distribution and out-of-distribution partitions.
  arXiv  Detail & Related papers  (2026-08-18T15:16:52Z)
• Induction Heads Interpolate N-Grams [29.939177622666858]
  Induction heads are attention circuits believed to underlie in-context learning in transformers.<n>We study transformers trained on order-$k$ Markov chains and identify two complementary smoothing mechanisms.<n>Our results bridge mechanistic interpretability of induction heads with classical statistical smoothing, revealing that transformers learn to regularize in-context estimation rather than simply count.
  arXiv  Detail & Related papers  (2026-07-02T22:19:26Z)
• Interpretable Probabilistic Medical Image Segmentation via Gaussian Process with Explicit Modelling of Annotation Bias and Variability [14.53294799919725]
  We propose a logit-space probabilistic segmentation framework that decomposes predictions into an image-dependent reference logit distribution and annotator specific perturbations parameterised by bias and variance.<n>We evaluate the method on a multi-annotator medical image dataset, which shows that explicitly modelling annotator specific perturbations improves uncertainty calibration.
  arXiv  Detail & Related papers  (2026-06-22T11:14:26Z)
• Transformers Are Born Biased: Structural Inductive Biases at Random Initialization and Their Practical Consequences [29.509249228044492]
  We show that randomly untrained models display extreme token preferences across random input sequences.<n>We show that extreme token preference arises from a contraction of token representations along a random seed-dependent direction.
  arXiv  Detail & Related papers  (2026-02-05T17:37:41Z)
• TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors [53.891337639229285]
  We introduce attentionLens, a novel formulation that captures the entire transformer as a single, input-dependent linear operator expressed through a high-order attention-interaction connection.<n>Our experiments demonstrate that the attention tensor can serve as a powerful foundation for developing tools aimed at interpretability and model understanding.
  arXiv  Detail & Related papers  (2026-01-25T19:21:25Z)
• From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformers [67.02076505996284]
  We study how the choice of pretraining data distribution steers a shallow transformer toward one behavior or the other.<n>Our results shed light on the algorithmic biases of pretrained transformers and offer conceptual guidelines for data-driven control of their learned behaviors.
  arXiv  Detail & Related papers  (2025-12-21T08:10:26Z)
• Entropy-Lens: The Information Signature of Transformer Computations [14.613982627206884]
  We study the evolution of token-level distributions directly in vocabulary space.<n>We compute the Shannon entropy of each intermediate predicted distribution, yielding one interpretable scalar per layer.<n>We introduce Entropy-Lens, a model-agnostic framework that extracts entropy profiles from frozen, off-the-shelf transformers.
  arXiv  Detail & Related papers  (2025-02-23T13:33:27Z)
• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Trained Transformers Learn Linear Models In-Context [39.56636898650966]
  Attention-based neural networks as transformers have demonstrated a remarkable ability to exhibit inattention learning (ICL) We show that when transformer training over random instances of linear regression problems, these models' predictions mimic nonlinear of ordinary squares.
  arXiv  Detail & Related papers  (2023-06-16T15:50:03Z)
• Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings [68.61185138897312]
  We show that a frozen transformer language model encodes strong positional information through the shrinkage of self-attention variance. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.
  arXiv  Detail & Related papers  (2023-05-23T01:03:40Z)
• Which Invariance Should We Transfer? A Causal Minimax Learning Approach [18.71316951734806]
  We present a comprehensive minimax analysis from a causal perspective. We propose an efficient algorithm to search for the subset with minimal worst-case risk. The effectiveness and efficiency of our methods are demonstrated on synthetic data and the diagnosis of Alzheimer's disease.
  arXiv  Detail & Related papers  (2021-07-05T09:07:29Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer [89.21584915290319]
  We propose a self-attention attribution method to interpret the information interactions inside Transformer. We show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.
  arXiv  Detail & Related papers  (2020-04-23T14:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.