Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity

• URL: http://arxiv.org/abs/2606.21158v1
• Date: Fri, 19 Jun 2026 06:49:09 GMT
• Title: Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
• Abstract summary: Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities.<n>We introduce DeadDirection Signatures (DDS), a family of cheap closed-form spectral readings of singular structure.<n>Each reads a network's activation matrix or per-sample-gradient FisherGram at a chosen layer, replacing the posterior chain spectral linear algebra.
• Score: 5.67642958049511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Singular learning theory characterises the complexity of a deep network through the geometry of its loss singularities. The local learning coefficient (LLC), the standard estimator of Watanabe's real log canonical threshold (RLCT, $λ$), reads this geometry as an integrated Bayesian scalar through SGLD, which needs per-task calibration and $10^4$-$10^6$ forward-backward passes per checkpoint. We introduce Dead-Direction Signatures (DDS), a family of cheap closed-form spectral readings of singular structure: each reads a network's activation matrix or per-sample-gradient Fisher-Gram at a chosen layer, replacing the SGLD posterior chain with spectral linear algebra. The readings rest on a dead-direction framework that predicts a structural correlation between activation- and Fisher-side spectra at any singular minimum, and a rank-multiplicative volume identity that single-eigenvalue monitors cannot produce: the active-volume $\log\det^{+}(G)$ slope counts the dead directions, tracking the rank-deficit $r$ across $r \in \{1,2,3,4\}$ (slope ratios $2.0, 3.1, 4.0$ at $r{=}2,3,4$ against the predicted $2,3,4$), where the smallest eigenvalue is rank-blind. On reduced-rank regression with closed-form $λ$, calibrated LLC recovers $λ$ at $99\%$ mean and the DDS observables rank-track it at the framework-predicted sign; on a non-linear modular-addition transformer DDS separates $d_{\mathrm{model}}$ across eighteen orders of magnitude where calibrated LLC at the protocol budget is rank-flat. Complementary to LLC's integrated posterior reading, DDS gives a directional, layer-local handle on a network's dead directions, read in closed form from its activation and gradient spectra.

Related papers

• Dead Directions: Geometric Singular Learning [0.0]
  Singular learning theory and information geometry have studied the same parameter spaces in mostly separate vocabularies.<n>We bridge them through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates.<n>A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold.<n>We then lift this rate to a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates.
  arXiv  Detail & Related papers  (2026-06-04T09:54:08Z)
• Representation Without Reward: A JEPA Audit for LLM Fine-Tuning [1.2691047660244335]
  Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs.<n>For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head emphand improve the decoded task metric.<n>We test that requirement under a fixed Llama-3.2-1B-Instruct LoRA harness on natural-language-to-regex generation.
  arXiv  Detail & Related papers  (2026-05-14T20:27:32Z)
• Geometric and Spectral Alignment for Deep Neural Network I [12.227949990332363]
  We prove deterministic quotient-geometric estimates for singular spectra of Frobenius-normalized layer factors.<n>Full-rank factors are mapped from $mathrmGL(d)$ to the positive cone by $Amapsto Atop A$, then to ordered eigenvalue data.<n>We prove approximate-power-law and metric-chart versions, converse lower bounds, Fisher--KL/Bures action estimates, and near-identity expansions for normalized residual chains.
  arXiv  Detail & Related papers  (2026-05-04T00:07:24Z)
• Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales [0.0]
  We show that transformer training trajectories evolve within only a few coherent directions.<n>In companion work, the same spectral geometry provides early-warning signals of grokking.
  arXiv  Detail & Related papers  (2026-03-14T04:46:05Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets [9.94514344279733]
  Arithmetic-Mean $mu$P constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale.<n>We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $etastar(L)propto L-3/2$; with zero padding, boundary effects are constant-level as $Ngg k$.
  arXiv  Detail & Related papers  (2025-10-05T19:22:50Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Emergence and scaling laws in SGD learning of shallow neural networks [64.48316762675141]
  We study the complexity of online gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data.<n>We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective.
  arXiv  Detail & Related papers  (2025-04-28T16:58:55Z)
• Settling the Sample Complexity of Model-Based Offline Reinforcement Learning [50.5790774201146]
  offline reinforcement learning (RL) learns using pre-collected data without further exploration. Prior algorithms or analyses either suffer from suboptimal sample complexities or incur high burn-in cost to reach sample optimality. We demonstrate that the model-based (or "plug-in") approach achieves minimax-optimal sample complexity without burn-in cost.
  arXiv  Detail & Related papers  (2022-04-11T17:26:19Z)
• Differentially Private Exploration in Reinforcement Learning with Linear Representation [102.17246636801649]
  We first consider the setting of linear-mixture MDPs (Ayoub et al., 2020) (a.k.a. model-based setting) and provide an unified framework for analyzing joint and local differential private (DP) exploration. We further study privacy-preserving exploration in linear MDPs (Jin et al., 2020) (a.k.a. model-free setting) where we provide a $widetildeO(sqrtK/epsilon)$ regret bound for $(epsilon,delta)
  arXiv  Detail & Related papers  (2021-12-02T19:59:50Z)
• On the Global Convergence of Training Deep Linear ResNets [104.76256863926629]
  We study the convergence of gradient descent (GD) and gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets) We prove that for training deep residual networks with certain linear transformations at input and output layers, both GD and SGD can converge to the global minimum of the training loss.
  arXiv  Detail & Related papers  (2020-03-02T18:34:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.