Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m

• URL: http://arxiv.org/abs/2605.24577v1
• Date: Sat, 23 May 2026 13:37:59 GMT
• Title: Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
• Abstract summary: Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation.<n>We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates.<n>The phenomenon is invisible to the standard SAE metric.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Independently trained transformers compute the same function in residual-stream bases that differ by a uniform random rotation on $\mathrm{SO}(d_{\mathrm{model}})$. We call this phenomenon polymorphism: same function, mutually unintelligible interior coordinates. One matrix multiplication per model pair removes it: an orthogonal Procrustes fit on a single batch of activations transfers sparse-autoencoder feature dictionaries and steering vectors between independently trained models, with no retraining. The phenomenon is invisible to the standard SAE universality metric. Decoder-column cosine similarity matches across seeds at 98%, the SAE-universality headline number, while an SAE trained on one seed reconstructs another seed's activations at negative explained variance, worse than predicting the constant mean. The decoder columns align; the encoder reads from a rotated frame. A single Procrustes rotation $R$ restores reconstruction to within 0.025 EV of the within-seed ceiling at every internal site. $R$ is Haar-distributed: $\|R - I\|_F$ matches the random-orthogonal prediction $\sqrt{2 d_{\mathrm{model}}}$ to 0.1% at $d_{\mathrm{model}} = 512$, and a Kolmogorov-Smirnov test of $R$'s eigenvalue spectrum against Haar $\mathrm{SO}(d_{\mathrm{model}})$ returns $p \approx 1.000$ pooled and per-pair. Diff-of-means steering vectors transfer in three regimes by alignment with $R$'s invariant subspace: clean when pinned by shared output weights, partial when overlapping the rotated subspace, inverted otherwise. With no shared I/O (Pythia), all three collapse to universally inverted. The same rotation account holds across training checkpoints within a single run. Validated on a 104k-parameter Dyck-3 transformer and nine independently-trained Pythia-70m seeds on The Pile, via a pre-registered four-bar operational framework. Frontier-scale (10B+) replication remains open.

Related papers

• $\ ext{GSF-}χ$: Global Stereochemical Fields for Chiral Graph Transformers [49.85093436361195]
  We introduce GSF-$$, a graph transformer in which stereogenic units modulate all pairwise interactions.<n>A $C$ projection separates mirror-even ECD peak counts and positions from mirror-odd peak signs.<n>GSF-$$ leads every central-ECD output and improves axial Rotation and Symbol by $12.6%$ and $7.9%$ over the strongest baseline.
  arXiv  Detail & Related papers  (2026-09-11T07:41:20Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group [6.230579198456525]
  A latent world model built from an equivariant encoder $E$ and an equivariant predictor $f$ inherits a provable symmetry of its training loss.<n>We verify this end-to-end at laptop scale (CPU/MPS, fully seeded)
  arXiv  Detail & Related papers  (2026-06-02T01:20:24Z)
• Hidden-State Privacy Has an Empty Middle [51.56484100374058]
  We prove a complementary Fisher-ball lower bound: every full-rank Gaussian release at $O(1)$ Fisher utility admits a direction whose Mahalanobis signal grows linearly in hidden width.<n>A split-memory transformer trained from scratch reaches $G_mathrmMah in [20, 33]$ at 90M and maintains a $6$--$24times$ advantage over same-budget GPT baselines from 30M to 1B at a fixed-token language loss penalty.
  arXiv  Detail & Related papers  (2026-05-21T20:12:09Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative [1.5718921092089344]
  Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks.<n>Under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length.<n>We argue that length-invariant metrics such as Centered Kernel Alignment should be the default for cross-representation comparisons.
  arXiv  Detail & Related papers  (2026-05-08T06:48:34Z)
• The E$Δ$-MHC-Geo Transformer: Adaptive Geodesic Operations with Guaranteed Orthogonality [2.28438857884398]
  E$$-MHC-Geo Transformer is a novel architecture that unifies Manifold-Constrained Hyper-Connections (mHC), Deep Delta Learning (DDL), and the Cayley transform.
  arXiv  Detail & Related papers  (2026-05-07T11:37:23Z)
• Dimensional Criticality at Grokking Across MLPs and Transformers [2.652953665748039]
  Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems.<n>We introduce textbfTDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe.<n>Across Transformers trained on modular addition and generalizations trained on XOR, we discover a localized crossing of the diffusion baseline $D=1$.
  arXiv  Detail & Related papers  (2026-04-06T13:43:20Z)
• Anisotropic local law for non-separable sample covariance matrices [10.181748307494608]
  We establish local laws for sample covariance matrices $K = N-1sum_i=1N g_ig_ig_i*$ where the random vectors $g_1, ldots, g_N in Rn$ are independent with common covariance $$.<n>We discuss several classes of non-separable examples satisfying our assumptions, including conditionally mean-zero distributions, the random features model $g = (Xw)$ arising in machine learning, and Gaussian measures with
  arXiv  Detail & Related papers  (2026-02-20T03:28:51Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Statistical Learning under Heterogeneous Distribution Shift [71.8393170225794]
  Ground-truth predictor is additive $mathbbE[mathbfz mid mathbfx,mathbfy] = f_star(mathbfx) +g_star(mathbfy)$.
  arXiv  Detail & Related papers  (2023-02-27T16:34:21Z)
• On Submodular Contextual Bandits [92.45432756301231]
  We consider the problem of contextual bandits where actions are subsets of a ground set and mean rewards are modeled by an unknown monotone submodular function. We show that our algorithm efficiently randomizes around local optima of estimated functions according to the Inverse Gap Weighting strategy.
  arXiv  Detail & Related papers  (2021-12-03T21:42:33Z)
• Variance-Aware Confidence Set: Variance-Dependent Bound for Linear Bandits and Horizon-Free Bound for Linear Mixture MDP [76.94328400919836]
  We show how to construct variance-aware confidence sets for linear bandits and linear mixture Decision Process (MDP) For linear bandits, we obtain an $widetildeO(mathrmpoly(d)sqrt1 + sum_i=1Ksigma_i2) regret bound, where $d is the feature dimension. For linear mixture MDP, we obtain an $widetildeO(mathrmpoly(d)sqrtK)$ regret bound, where
  arXiv  Detail & Related papers  (2021-01-29T18:57:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.