The Laplacian Mechanism Improves Transformers by Reshaping Token Geometry

• URL: http://arxiv.org/abs/2602.09297v1
• Date: Tue, 10 Feb 2026 00:27:45 GMT
• Title: The Laplacian Mechanism Improves Transformers by Reshaping Token Geometry
• Authors: Yuchong Zhang, Vardan Papyan,
• Abstract summary: We show that incorporating the Laplacian mechanism into transformers induces consistent improvements across benchmarks in computer vision and language.<n>Our investigation shows that the Laplacian mechanism reshapes token embeddings toward a geometry of maximal separability.
• Score: 15.311893064721856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers leverage attention, the residual connection, and layer normalization to control the variance of token representations. We propose to modify attention into a Laplacian mechanism that gives the model more direct control over token variance. We conjecture that this helps transformers achieve the ideal token geometry. To investigate our conjecture, we first show that incorporating the Laplacian mechanism into transformers induces consistent improvements across benchmarks in computer vision and language. Next, we study how the Laplacian mechanism impacts the geometry of token representations using various tools: 1) principal component analysis, 2) cosine similarity metric, 3) analysis of variance, and 4) Neural Collapse metrics. Our investigation shows that the Laplacian mechanism reshapes token embeddings toward a geometry of maximal separability: tokens collapse according to their classes, and the class means exhibit Neural Collapse.

Related papers

• Selective Induction Heads: How Transformers Select Causal Structures In Context [50.09964990342878]
  We introduce a novel framework that showcases transformers' ability to handle causal structures.<n>Our framework varies the causal structure through interleaved Markov chains with different lags while keeping the transition probabilities fixed.<n>This setting unveils the formation of Selective Induction Heads, a new circuit that endows transformers with the ability to select the correct causal structure in-context.
  arXiv  Detail & Related papers  (2025-09-09T23:13:41Z)
• Plain Transformers Can be Powerful Graph Learners [64.50059165186701]
  Researchers have attempted to migrate Transformers to graph learning, but most advanced Graph Transformers have strayed far from plain Transformers.<n>This work demonstrates that the plain Transformer architecture can be a powerful graph learner.
  arXiv  Detail & Related papers  (2025-04-17T02:06:50Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Clustering in pure-attention hardmax transformers and its role in sentiment analysis [0.0]
  We rigorously characterize the behavior of transformers with hardmax self-attention and normalization sublayers as the number of layers tends to infinity. We show that the transformer inputsally converge to a clustered equilibrium determined by special points called leaders. We then leverage this theoretical understanding to solve sentiment analysis problems from language processing using a fully interpretable transformer model.
  arXiv  Detail & Related papers  (2024-06-26T16:13:35Z)
• Token Transformation Matters: Towards Faithful Post-hoc Explanation for Vision Transformer [16.97186100288621]
  Vision Transformers extract visual information by representing regions as transformed tokens and integrating them via attention weights. Existing post-hoc explanation methods merely consider these attention weights, neglecting crucial information from the transformed tokens. We propose TokenTM, a novel post-hoc explanation method that utilizes our introduced measurement of token transformation effects.
  arXiv  Detail & Related papers  (2024-03-21T16:52:27Z)
• Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling [10.246977481606427]
  We study the mechanisms through which different components of Transformer, such as the dot-product self-attention, affect its expressive power. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads.
  arXiv  Detail & Related papers  (2024-02-01T11:43:13Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• What Makes for Good Tokenizers in Vision Transformer? [62.44987486771936]
  transformers are capable of extracting their pairwise relationships using self-attention. What makes for a good tokenizer has not been well understood in computer vision. Modulation across Tokens (MoTo) incorporates inter-token modeling capability through normalization. Regularization objective TokenProp is embraced in the standard training regime.
  arXiv  Detail & Related papers  (2022-12-21T15:51:43Z)
• Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers [52.468311268601056]
  This paper analyzes attention through the lens of convex duality. We derive equivalent finite-dimensional convex problems that are interpretable and solvable to global optimality. We show how self-attention networks implicitly cluster the tokens, based on their latent similarity.
  arXiv  Detail & Related papers  (2022-05-17T04:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.