Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment

• URL: http://arxiv.org/abs/2607.00603v1
• Date: Wed, 01 Jul 2026 08:29:36 GMT
• Title: Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment
• Abstract summary: We give a descent-free, alignment-free measurement of singular structure on trained networks.<n>A read recovers architecture-predicted order across constructed cells and trained networks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We give a descent-free, alignment-free measurement of singular structure on trained networks. At a single frozen checkpoint the read recovers the order $k$ of each dead direction from the directional-Fisher rate, the master invariant from which the per-direction learning coefficient $1/(2k)$ follows exactly, in whatever basis the optimizer left. The same read classifies each direction, separating a genuine singularity, whose order the architecture fixes, from a flat gauge symmetry; the directional-Fisher magnitude settles the cases the order cannot. A pluggable detector supplies the directions for transformer, convolutional, and normalisation layers. The read recovers the architecture-predicted order across constructed cells and trained networks, including a fine-tuned vision transformer whose dead structure is the LayerNorm-kernel gauge and a from-scratch one whose compressed MLP forms a node-death at its activation order. Where the singular structure enumerates, the per-direction orders assemble, through the typed intersection of the loci, into the global coefficient $(λ, m)$ matching the closed form. The method removes the canonical-alignment and descent preconditions of the underlying rate result, turning order-recovery into a deterministic, architecture-general reading. We then map its reach into the Watanabe triple: the order determines the universal singular fluctuation $ν(k)$, though a trained network's realized $ν$ falls below it as the live structure absorbs the dead direction's data fluctuation, and the multiplicity recovers from the dominant structure under a single-locus assumption.

Related papers

• Simplex Relaxation for Discrete Diffusion [58.33696160279345]
  We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable.<n>Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation.
  arXiv  Detail & Related papers  (2026-08-11T07:58:17Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity [5.67642958049511]
  Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities.<n>We introduce DeadDirection Signatures (DDS), a family of cheap closed-form spectral readings of singular structure.<n>Each reads a network's activation matrix or per-sample-gradient FisherGram at a chosen layer, replacing the posterior chain spectral linear algebra.
  arXiv  Detail & Related papers  (2026-06-19T06:49:09Z)
• Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scale [5.67642958049511]
  Inverse-scale direction $-1/|-1|$ of the LayerNorm affine is an exact kernel of the post-final-norm centred activation covariance.<n>It is read from the LN scale parameter alone, with no forward or backward pass and no eigensolve.
  arXiv  Detail & Related papers  (2026-06-17T18:28:37Z)
• Dead Directions: Geometric Singular Learning [0.0]
  Singular learning theory and information geometry have studied the same parameter spaces in mostly separate vocabularies.<n>We bridge them through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates.<n>A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold.<n>We then lift this rate to a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates.
  arXiv  Detail & Related papers  (2026-06-04T09:54:08Z)
• Learning Transferable Predictability Representations [2.1505812656718795]
  We study the problem of assigning a scalar score to a short trajectory window that reflects its position on an ordered continuum of predictability regimes.<n>We formalize this as ordinal estimation over a five-level predictability ladder and identify a structural source of cross-system ambiguity.<n>We propose the Gauge-Fixed Ordinal Network (GON), a temporal convolutional model trained with an anchor-and-variance objective that pins level-wise score means to shared target coordinates.
  arXiv  Detail & Related papers  (2026-05-28T21:38:49Z)
• Deep Delta Learning [91.75868893250662]
  We introduce Deep Delta Learning (DDL), a novel architecture that generalizes the standard residual connection.<n>We provide a spectral analysis of this operator, demonstrating that the gate $(mathbfX)$ enables dynamic between identity mapping, projection, and geometric reflection.<n>This unification empowers the network to explicitly control the spectrum of its layer-wise transition operator, enabling the modeling of complex, non-monotonic dynamics.
  arXiv  Detail & Related papers  (2026-01-01T18:11:38Z)
• Phase-space entropy at acquisition reflects downstream learnability [54.4100065023873]
  We propose an acquisition-level scalar $S_mathcal B$ based on instrument-resolved phase space.<n>We show theoretically that (S_mathcal B) correctly identifies the phase-space coherence of periodic sampling.<n>$|S_mathcal B|$ consistently ranks sampling geometries and predicts downstream reconstruction/recognition difficulty emphwithout training.
  arXiv  Detail & Related papers  (2025-12-22T10:03:51Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.