All Routes Lead to Collapse

• URL: http://arxiv.org/abs/2606.22325v1
• Date: Sun, 21 Jun 2026 03:59:13 GMT
• Title: All Routes Lead to Collapse
• Abstract summary: We show that attention sinks, representation collapse, and norm stratification are not transformer-specific pathologies.<n>We give a reframing identity: softmax attention is Boltzmann-weighted aggregation over Euclidean distances with constant key norms.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention sinks, representation collapse, and norm stratification are treated as transformer-specific pathologies. We show they are not specific to attention: they are what content-based routing does under a fixed similarity metric. We give a reframing identity: softmax attention is Boltzmann-weighted aggregation over Euclidean distances with constant key norms, so its score omits a $-\|k\|^2$ term and is blind to key magnitude. This predicts that any router whose metric is ill-matched to its representations should compensate, by concentrating its routing and collapsing the routed representations. We test it on routers that score and aggregate over different axes: softmax attention over tokens (nine pretrained transformers), graph attention over nodes, a selective state-space model and a recurrent mixer over time, and learned residuals over depth. All develop the same signature, and two within-model ablations show it is caused by the routing mechanism rather than by incidental dynamics. The form is contingent, set by the strength of the positional brake each router carries alongside its content score; we sweep that brake and move the onset across its whole range. The mechanism is not contingent, and it does not require norm stratification: a router with norm-normalized keys concentrates just the same. We do not claim these models implement Riemannian geometry; the geometric view is a diagnostic that names the inadequacy of the flat, norm-blind metric.

Related papers

• Dense Structural Compression of Transformers via Gauge-Correct Channel Removal [11.681900731333926]
  Inference energy per token drives the cost and carbon footprint of deployed transformers.<n>We develop a methodology to adapt structural complexity during training to maximize utility per unit compute.<n>Channel penalties drive entire tensor slices to zero to enable physical removal while preserving density and the network function.
  arXiv  Detail & Related papers  (2026-09-07T09:20:59Z)
• From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers [53.505136935517925]
  Self-attention is a ubiquitous primitive in modern sequence models, yet its operator-level geometry is only partially understood.<n>We prove that single-head attention (SHA) is exactly a connection propagation step with constant transport.<n>We find that trained Transformers across scales (from 124M to 8B) and structures (encoder/decoder) exhibit geometric structure consistent with our theory.
  arXiv  Detail & Related papers  (2026-07-12T09:44:54Z)
• Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention [1.7042264000899534]
  We show that the strongest previous-token heads are spectrally rotational under RoPE.<n>We also show that over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null.<n>Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution.
  arXiv  Detail & Related papers  (2026-07-07T10:17:30Z)
• Redesign Mixture-of-Experts Routers with Manifold Power Iteration [60.55368504665436]
  We propose a router redesign with Manifold Power Iteration (MPI)<n>MPI drives router rows to converge toward the principal singular directions of associated experts.<n>We show that MPI drives router rows to converge toward the principal singular directions of associated experts.
  arXiv  Detail & Related papers  (2026-06-10T17:57:36Z)
• Hybrid Neural World Models [1.0742675209112622]
  Neural surrogates promise large speedups over classical solvers for physical dynamics but fail silently at sharp dynamical events such as shocks, fronts, and contact.<n>We present a recipe for training and deploying multi-horizon surrogates in physical state space.<n>The recipe applies without modification across reaction-diffusion, compressible Euler, and rigid-body collision dynamics.
  arXiv  Detail & Related papers  (2026-05-27T11:17:08Z)
• How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Models [0.0]
  This paper localizes the policy routing mechanism in alignment-trained language models.<n>An intermediate-layer attention gate reads detected content and triggers deeper heads that boost the signal toward refusal.
  arXiv  Detail & Related papers  (2026-04-06T03:20:37Z)
• The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology [0.0]
  We study grokking - delayed generalization in Transformers trained on cyclic modular addition (Zp)<n>We identify two independent structural factors in standard Transformers: representational magnitude and data-dependent attention routing.
  arXiv  Detail & Related papers  (2026-03-05T14:41:01Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It [5.89889361990138]
  We argue that existing sharpness measures fail for transformers because they have much richer symmetries in their attention mechanism.<n>We propose a fully general notion of sharpness, in terms of a geodesic ball on the symmetry-corrected quotient manifold.<n>We show that our geodesic sharpness reveals strong correlation for real-world transformers on both text and image classification tasks.
  arXiv  Detail & Related papers  (2025-05-08T16:51:03Z)
• Gramformer: Learning Crowd Counting via Graph-Modulated Transformer [68.26599222077466]
  Gramformer is a graph-modulated transformer to enhance the network by adjusting the attention and input node features respectively. A feature-based encoding is proposed to discover the centrality positions or importance of nodes. Experiments on four challenging crowd counting datasets have validated the competitiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-01-08T13:01:54Z)
• Attention Map Guided Transformer Pruning for Edge Device [98.42178656762114]
  Vision transformer (ViT) has achieved promising success in both holistic and occluded person re-identification (Re-ID) tasks. We propose a novel attention map guided (AMG) transformer pruning method, which removes both redundant tokens and heads. Comprehensive experiments on Occluded DukeMTMC and Market-1501 demonstrate the effectiveness of our proposals.
  arXiv  Detail & Related papers  (2023-04-04T01:51:53Z)
• HDGT: Heterogeneous Driving Graph Transformer for Multi-Agent Trajectory Prediction via Scene Encoding [76.9165845362574]
  We propose a backbone modelling the driving scene as a heterogeneous graph with different types of nodes and edges. For spatial relation encoding, the coordinates of the node as well as its in-edges are in the local node-centric coordinate system. Experimental results show that HDGT achieves state-of-the-art performance for the task of trajectory prediction.
  arXiv  Detail & Related papers  (2022-04-30T07:08:30Z)
• Do Generative Models Know Disentanglement? Contrastive Learning is All You Need [59.033559925639075]
  We propose an unsupervised and model-agnostic method: Disentanglement via Contrast (DisCo) in the Variation Space. DisCo achieves the state-of-the-art disentanglement given pretrained non-disentangled generative models, including GAN, VAE, and Flow.
  arXiv  Detail & Related papers  (2021-02-21T08:01:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.