Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space

• URL: http://arxiv.org/abs/2605.16600v1
• Date: Fri, 15 May 2026 20:00:59 GMT
• Title: Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight space
• Abstract summary: Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces.<n>We characterise this asymmetry with a relative-subspace-fraction probe that tracks how weight deltas align with residual-stream activation subspaces.
• Score: 2.2940141855172036
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces. We characterise this asymmetry with a relative-subspace-fraction probe that tracks how weight deltas align with residual-stream activation subspaces and with the prediction subspace defined by the unembedding. Alignment deltas concentrate in the read pathway ($W_Q$, $W_K$), along principal directions of attention-input activations, while remaining near-isotropic in the write pathway ($W_O$, $W_2$) relative to the prediction subspace. We explain this pattern through anisotropic gradient accumulation: updates to a matrix $W$ are sums of outer products $δ_t a_t^\top$, and inherit directional structure from whichever side has concentrated covariance. For read-pathway matrices, this side is the input activation $a_t$, whose covariance is spiked in trained transformers and therefore produces objective-agnostic concentration. For write-pathway matrices, the relevant side is the upstream gradient $δ_t$, whose anisotropy depends on the loss. Cross-entropy supplies the canonical sharp per-sample signal, inducing write-pathway prediction geometry during pretraining; alignment objectives typically add little further write-side concentration. We support this explanation with a within-checkpoint trajectory, a graded contrastive-objective control, and a closed-form rank-1 intervention with matched direction controls, providing causal evidence for the proposed weight-space geometry.

Related papers

• Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale [5.67642958049511]
  Inverse-scale direction $-1/|-1|$ of the LayerNorm affine is an exact kernel of the post-final-norm centred activation covariance.<n>It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve.
  arXiv  Detail & Related papers  (2026-06-17T18:28:37Z)
• How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models [6.758792133077052]
  We derive exact characterizations of the limiting eigenvalue distribution, outlier eigenvalues, and eigenvector alignment with the hidden signal.<n>We show that parameter-free causal self-attention with $/d$ score scaling yields deterministic harmonic weights that improve signal recovery over mean pooling.
  arXiv  Detail & Related papers  (2026-05-07T18:28:01Z)
• Symmetric Equilibrium Propagation for Thermodynamic Diffusion Training [0.0]
  We show that a bilinearly-coupled analog substrate can physically realize dynamics at a projected three-to-four orders of magnitude energy advantage over digital inference.<n>We resolve whether the emphtraining loop can be closed on the same substrate without routing gradients through an external digital accelerator.
  arXiv  Detail & Related papers  (2026-04-26T17:05:05Z)
• OLion: Approaching the Hadamard Ideal by Intersecting Spectral and $\ell_{\infty}$ Implicit Biases [29.60546958677364]
  nameA combines spectral control from update directions with coordinate control from sign updates.<n>We prove convergence under a mild, empirically verified diagonal-isotropy assumption.<n>nameA matches or outperforms AdamW and Muon under comparable tuning while using only momentum-level state.
  arXiv  Detail & Related papers  (2026-02-01T08:59:45Z)
• Phase-space entropy at acquisition reflects downstream learnability [54.4100065023873]
  We propose an acquisition-level scalar $S_mathcal B$ based on instrument-resolved phase space.<n>We show theoretically that (S_mathcal B) correctly identifies the phase-space coherence of periodic sampling.<n>$|S_mathcal B|$ consistently ranks sampling geometries and predicts downstream reconstruction/recognition difficulty emphwithout training.
  arXiv  Detail & Related papers  (2025-12-22T10:03:51Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Mesoscopic Fluctuations and Multifractality at and across Measurement-Induced Phase Transition [46.176861415532095]
  We explore statistical fluctuations over the ensemble of quantum trajectories in a model of two-dimensional free fermions.<n>Our results exhibit a remarkable analogy to Anderson localization, with $G_AB$ corresponding to two-terminal conductance.<n>Our findings lay the groundwork for mesoscopic theory of monitored systems, paving the way for various extensions.
  arXiv  Detail & Related papers  (2025-07-15T13:44:14Z)
• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Superdiffusion in random two dimensional system with time-reversal symmetry and long-range hopping [45.873301228345696]
  localization problem in the crossover regime for the dimension $d=2$ and hopping $V(r) propto r-2$ is not resolved yet. We show that for the hopping determined by two-dimensional anisotropic dipole-dipole interactions there exist two distinguishable phases at weak and strong disorder.
  arXiv  Detail & Related papers  (2022-05-29T16:53:20Z)
• Light transfer transitions beyond higher-order exceptional points in parity-time and anti-parity-time symmetric waveguide arrays [7.21172766653517]
  Two non-Hermitian arrays are proposed for investigating light transfer dynamics based on $N$th-order exceptional points (EPs) The $mathcalPT$-symmetric array supports two $N$th-order EPs separating an unbroken and a broken phase with real and imaginary eignvalues, respectively. The anti-$mathcalPT$-symmetric array supports also two $N$th-order EPs separating an unbroken and a broken phase, which refer however to imaginary and real eignvalues, respectively.
  arXiv  Detail & Related papers  (2022-05-16T10:51:38Z)
• Analytic Signal Phase in $N-D$ by Linear Symmetry Tensor--fingerprint modeling [69.35569554213679]
  We show that the Analytic Signal phase, and its gradient have a hitherto unstudied discontinuity in $2-D $ and higher dimensions. This shortcoming can result in severe artifacts whereas the problem does not exist in $1-D $ signals. We suggest the use of Linear Symmetry phase, relying on more than one set of Gabor filters, but with a negligible computational add-on.
  arXiv  Detail & Related papers  (2020-05-16T21:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.