Stream separation improves Bregman conditioning in transformers

• URL: http://arxiv.org/abs/2603.21317v1
• Date: Sun, 22 Mar 2026 16:55:57 GMT
• Title: Stream separation improves Bregman conditioning in transformers
• Abstract summary: Linear methods for steering transformer representations implicitly assume the geometry of representation space is Euclidean.<n>Park et al. showed that softmax induces a curved Bregman geometry whose metric tensor is the Hessian of the log-normalizer, $H() = Cov[ | ]$.<n>We measure this Hessian at intermediate layers in a controlled 2x2 design crossing stream separation with per-layer supervision.
• Score: 4.7718339202518685
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Linear methods for steering transformer representations, including probing, activation engineering, and concept erasure, implicitly assume the geometry of representation space is Euclidean. Park et al. [Park et al., 2026] showed that softmax induces a curved Bregman geometry whose metric tensor is the Hessian of the log-normalizer, $H(λ) = Cov[γ | λ]$. Ignoring this curvature causes Euclidean steering to leak probability mass to unintended tokens. Their analysis applies at the output layer. We measure this Hessian at intermediate layers in a controlled 2x2 design crossing stream separation with per-layer supervision (vocabulary decoding loss at each layer), all at matched vocabulary and parameter count. In standard single-stream transformers, H is severely degenerate at intermediate layers (effective rank 8 in 516 dimensions). Stream separation improves conditioning by up to 22 in effective rank, even without auxiliary supervision. Per-layer supervision helps, but less. The cosine similarity between primal and dual concept directions predicts per-layer steering effectiveness on downstream tasks, with a threshold near 0.3. These results bear on the reliability of linear safety interventions, which depend on the geometry being well-conditioned at the layer where they are applied.

Related papers

• 3D Euler-Angle Orientation Control for Two-Ray Fading Mitigation in Maritime Air-to-Sea Communications [6.741942263052468]
  Maritime Air-to-Sea links are dominated by a line-of-sight ray and a sea-surface reflected ray whose destructive combination produces deep fades.<n>Existing mitigation strategies optimize Unmanned Aerial Vehicle position or trajectory but leave attitude unexploited.<n>This paper treats the full three dimensional attitude as a physical-layer control variable that shapes the two-ray interference through antenna phase-center displacement.
  arXiv  Detail & Related papers  (2026-09-10T12:42:04Z)
• Geometry-Guided Layerwise FFN Width Allocation in Transformers [4.109803241412197]
  Feed-forward networks (FFNs) account for a large fraction of Transformer parameters, yet their hidden width is usually constant across depth.<n>We ask whether this capacity can instead be allocated from a forward-pass measurement of layer behavior.<n>We view each FFN as transporting a cloud of token representations and quantify the induced geometric change using correspondence-preserving shift, Gromov-Wasserstein distortion, and degree-one persistent homology under raw and scale-normalized metrics.
  arXiv  Detail & Related papers  (2026-08-03T11:08:58Z)
• Attractor Geometry Determines the Identifiability Limits of System Discovery [0.764671395172401]
  We show that a single number, $_min(M)$, sets the identifiability ceiling for both sparse regression (SINDy) and evolutionary regression (PySR)<n>We also introduce Soft F1, a coefficient-weighted structural metric that resolves performance differences invisible to binary-success and predictive scores.
  arXiv  Detail & Related papers  (2026-07-20T20:19:19Z)
• CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry [34.36337082439934]
  Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers add little value.<n>We present two efficiency methods that exploit this asymmetry.<n> CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8.6% and increasing throughput by 9.4%.
  arXiv  Detail & Related papers  (2026-06-25T02:25:00Z)
• Taming Curvature: Architecture Warm-Up for Stable Transformer Training [52.6838951211596]
  We introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power with Hessian-vector products.<n>We find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth.<n> Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities.
  arXiv  Detail & Related papers  (2026-06-15T14:16:56Z)
• Hierarchical Transformer Preconditioning for Interactive Physics Simulation [28.137076331332413]
  Hierarchical Transformer Preconditioner, a neural preconditioner anchored to a weak-admissibility H-matrix partition.<n>Network models the inverse through low-rank far-field factors.<n>Uses highway connections to propagate context across depth sparse.
  arXiv  Detail & Related papers  (2026-05-13T11:02:27Z)
• Residual Stream Duality in Modern Transformer Architectures [9.910562011343009]
  Recent work has made clear that the residual pathway is not mere optimization plumbing; it is part of the model's representational machinery.<n>We argue that the cleanest way to organize this design space is through a two-axis view of the Transformer.
  arXiv  Detail & Related papers  (2026-03-17T00:56:29Z)
• Latent attention on masked patches for flow reconstruction [8.69419238669827]
  We introduce the Latent Attention on Masked Patches (LAMP) model, an interpretable regression-based modified vision transformer for masked flow reconstruction.<n>We show that the LAMP accurately reconstructs the full flow field from a 90%-masked and noisy input, across signal-to-noise ratios between 10 and 30,dB.
  arXiv  Detail & Related papers  (2026-03-02T16:12:40Z)
• Gated Removal of Normalization in Transformers Enables Stable Training and Efficient Inference [2.1665689529884697]
  We introduce TaperNorm, a drop-in replacement for RMSNorm/LayerNorm that behaves exactly like the standard normalizer early in training.<n>A single global gate is held at $g=1$ during gate warmup, used to calibrate the scaling branch via EMAs, and then cosine-decayed to $g=0$.<n>On an efficiency microbenchmark, folding internal scalings yields up to $1.22times$ higher throughput in last-token logits mode.
  arXiv  Detail & Related papers  (2026-02-11T01:40:34Z)
• How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias [48.9399496805422]
  We focus on two representative tasks in the category of regular language recognition, known as even pairs' and parity check'<n>Our goal is to explore how a one-layer transformer, consisting of an attention layer followed by a linear layer, learns to solve these tasks.
  arXiv  Detail & Related papers  (2025-05-02T00:07:35Z)
• Parallelizing Linear Transformers with the Delta Rule over Sequence Length [49.88826673324244]
  This work describes a hardware-efficient algorithm for training linear transformers with the delta rule.<n>We train a 1.3B model for 100B tokens and find that it outperforms recent linear-time baselines.
  arXiv  Detail & Related papers  (2024-06-10T17:24:42Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generation [64.92152574895111]
  We propose a simple Orthogonal Jacobian Regularization (OroJaR) to encourage deep generative model to learn disentangled representations. Our method is effective in disentangled and controllable image generation, and performs favorably against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-17T15:01:46Z)
• Orthogonalizing Convolutional Layers with the Cayley Transform [83.73855414030646]
  We propose and evaluate an alternative approach to parameterize convolutional layers that are constrained to be orthogonal. We show that our method indeed preserves orthogonality to a high degree even for large convolutions.
  arXiv  Detail & Related papers  (2021-04-14T23:54:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.