Tracing distinguishability through transformer processing with stochastic LayerNorm

• URL: http://arxiv.org/abs/2608.30720v1
• Date: Mon, 31 Aug 2026 12:57:38 GMT
• Title: Tracing distinguishability through transformer processing with stochastic LayerNorm
• Abstract summary: We give representations volume, turning similarity into statistical distinguishability.<n>We realize this idea in pretrained transformers through a light-touch modification to LayerNorm.<n>These results establish distinguishability as a functionally grounded lens on transformer computation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representational similarity is foundational to analyses of deep networks, yet distances between point-valued representations are not intrinsically tied to downstream function: nearby states may produce different behaviors, while distant states may behave similarly. We instead give representations volume, turning similarity into statistical distinguishability. Overlapping stochastic representations necessarily induce overlapping downstream distributions, grounding latent comparison in model function and bringing it under information-theoretic tools such as the data-processing inequality. We realize this idea in pretrained transformers through a light-touch modification to LayerNorm: at each residual-stream read, we normalize the state, add isotropic Gaussian noise, and renormalize. During distillation fine-tuning, one learned allocation parameter per residual-stream read distributes a fixed global rate budget across the processing stack. The resulting model can be viewed as transformer blocks reading the residual stream with learned finite precision under a shared global rate budget. Using the Bhattacharyya coefficient, we trace which counterfactual distinctions are preserved through MLP blocks or selectively exposed to the query, key, and value computations of individual attention heads. Experiments on ViT-S and GPT-2 small reveal the depthwise propagation of continuous visual perturbations and head-specific sensitivity to token distinctions aligned with known attention motifs. These results establish distinguishability as a functionally grounded lens on transformer computation that complements existing interpretability approaches.

Related papers

• Interior interpretability with attention rollout: contraction and propagation profiles in Transformers [17.72586636966418]
  We introduce emphinterior interpretability, a propagation-based perspective on internal model organization.<n>We interpret rollout as a row-stochastic operator encoding attention-mediated propagation between feature tokens.
  arXiv  Detail & Related papers  (2026-07-24T14:51:28Z)
• DIFF-ERO: A Conformance-Aware Loss for Deep Learning in Process Mining [8.304025820176394]
  We introduce DIFF-ERO, a conformance-aware loss function for deep learning models on process data.<n>DIFF-ERO is a differentiable formulation of entropy-based conformance that incorporates control-flow information during training.<n>We show improved predictive performance where structure matters most while maintaining parity elsewhere.
  arXiv  Detail & Related papers  (2026-06-12T09:06:05Z)
• Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers [9.617245548268437]
  Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers.<n>This creates a subtle misalignment: residual connections tie activations to the current token, while supervision targets the next token.<n>We propose a lightweight residual-path mitigation based on residual attenuation, implemented either as a fixed-layer intervention or as a learnable gating mechanism.
  arXiv  Detail & Related papers  (2026-02-16T14:04:42Z)
• FreSca: Scaling in Frequency Space Enhances Diffusion Models [55.75504192166779]
  This paper explores frequency-based control within latent diffusion models.<n>We introduce FreSca, a novel framework that decomposes noise difference into low- and high-frequency components.<n>FreSca operates without any model retraining or architectural change, offering model- and task-agnostic control.
  arXiv  Detail & Related papers  (2025-04-02T22:03:11Z)
• Unsupervised Representation Learning from Sparse Transformation Analysis [79.94858534887801]
  We propose to learn representations from sequence data by factorizing the transformations of the latent variables into sparse components. Input data are first encoded as distributions of latent activations and subsequently transformed using a probability flow model.
  arXiv  Detail & Related papers  (2024-10-07T23:53:25Z)
• Variance-Reducing Couplings for Random Features [57.73648780299374]
  Random features (RFs) are a popular technique to scale up kernel methods in machine learning. We find couplings to improve RFs defined on both Euclidean and discrete input spaces. We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
  arXiv  Detail & Related papers  (2024-05-26T12:25:09Z)
• Score-based Continuous-time Discrete Diffusion Models [102.65769839899315]
  We extend diffusion models to discrete variables by introducing a Markov jump process where the reverse process denoises via a continuous-time Markov chain. We show that an unbiased estimator can be obtained via simple matching the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.
  arXiv  Detail & Related papers  (2022-11-30T05:33:29Z)
• Treatment Learning Causal Transformer for Noisy Image Classification [62.639851972495094]
  In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy. Motivated from causal variational inference, we propose a transformer-based architecture, that uses a latent generative model to estimate robust feature representations for noise image classification. We also create new noisy image datasets incorporating a wide range of noise factors for performance benchmarking.
  arXiv  Detail & Related papers  (2022-03-29T13:07:53Z)
• Robust Correction of Sampling Bias Using Cumulative Distribution Functions [19.551668880584973]
  Varying domains and biased datasets can lead to differences between the training and the target distributions. Current approaches for alleviating this often rely on estimating the ratio of training and target probability density functions.
  arXiv  Detail & Related papers  (2020-10-23T22:13:00Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Latent Transformations for Discrete-Data Normalising Flows [15.005894753472894]
  We present an unbiased alternative where rather than deterministically parameterising one transformation, we predict a distribution over latent transformations. With transformations, the marginal likelihood of the data is differentiable and gradient-based learning is possible via score function estimation. We observe great challenges with both deterministic proxy gradients and unbiased score function estimation.
  arXiv  Detail & Related papers  (2020-06-11T11:41:28Z)
• Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation [72.40827239394565]
  We propose to compute features only at sparsely sampled locations. We then densely reconstruct the feature map with an efficient procedure. The presented network is experimentally shown to save substantial computation while maintaining accuracy over a variety of computer vision tasks.
  arXiv  Detail & Related papers  (2020-03-19T15:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.