No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

• URL: http://arxiv.org/abs/2607.05872v1
• Date: Tue, 07 Jul 2026 06:06:04 GMT
• Title: No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
• Abstract summary: Memory-efficients such as GaLore train large language models by projecting gradients onto a rank-r subspace.<n>We show that beyond a small reproducible core, there is no such object.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Memory-efficient optimizers such as GaLore train large language models by projecting gradients onto a rank-r subspace recomputed every T steps, assuming this subspace is a slowly drifting object that can be tracked. We show that beyond a small reproducible core, there is no such object. Two estimates of the top-r subspace computed at the same step from disjoint minibatches disagree as much as estimates computed T steps apart (0.73 vs 0.74 of the maximal chordal distance sqrt(2r), at Pythia-160M with r=128): the apparent rotation at each refresh is dominated by estimator noise. This holds across four model families in three architecture classes from 70M to 6.9B parameters, strengthening with scale, and more weakly in a vision transformer. Only ~39 of 128 directions are reproducible across minibatches, and averaging cannot recover the rest: under N-fold averaging the gradient's spectral tail shrinks as N^(-1/4) rather than the N^(-1/2) of pure noise, so no averaging budget makes the subspace well defined. What helps instead follows from treating each refresh as a change of coordinates for Adam's state. Carrying the second moment blindly is provably about (r-k*)/2 worse than the best rotation-blind estimator, while the first moment transports exactly through the rotation, the optimal linear map under isotropic gradients and the rule LDAdam uses. At 1B over 40k steps (3 seeds), full LDAdam reaches 18.7 perplexity at beta2=0.999, beating untransported GaLore after its best beta2 fix (19.3); shortening the second-moment memory to beta2=0.99 helps the refreshing optimizers, though for canonical GaLore the effect is small and a full-rank control reverses it. One measurable fact, subspace non-identifiability, clarifies why GaLore works, which patches work, and what to check before trusting a low-rank assumption: the reproducible rank k*.

Related papers

• EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale [9.562123195246393]
  EGGROLL makes evolution strategies practical for LLMs by replacing dense Gaussian weight perturbations with low-rank Gaussian products.<n>We analyze the error of its finite-population estimator.<n>LOO-ROLL, a leave-one-out estimator, preserves the finite-rank population field while replacing EGGROLL's two antithetic evaluations per direction by one.
  arXiv  Detail & Related papers  (2026-09-10T01:58:02Z)
• Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't [0.0]
  Wecharacterize the 1.4B->410M conversion in the Pythia family end to end.<n>Weight residuals are statistically indistinguishable from noiseunder shuffle controls.<n>Compensationis a token-efficient, low-budget win rather than a universal one.
  arXiv  Detail & Related papers  (2026-08-03T19:44:08Z)
• Level-Crossing Density as a Mesh-Free High-Frequency Auxiliary Loss for Implicit Neural Representations [0.0]
  We derive smooth Monte-Carlo estimators for a continuous neural field using no grid, no complex, no persistence.<n>The estimator is accurate to 1-3% against exact topology in 2D and 3D, and costs about 3 ms per iteration.<n>In 3D neural-SDF fitting, however, a failure mode we believe general to any sampled soft topology objective appears.
  arXiv  Detail & Related papers  (2026-07-07T04:23:23Z)
• Dead-Direction Conditioners: Gauge-Equivariant Preconditioning for Deep Networks [0.0]
  A deep network's loss is invariant to continuous symmetries of its parameters.<n>We build DDC, a Dead-Direction Conditioner that conditions the orbit decomposition of a $G$-invariant metric.<n>On a language model trained past the point of fit, DDCAdam resists the over-training collapse AdamW falls into, holding a validation-train loss gap of 0.67 against 5.88, and reads the dead-direction rate in 32 of 65 layer-by-observable cells.
  arXiv  Detail & Related papers  (2026-06-28T03:44:59Z)
• First-Passage Prediction of Grokking Delay: ACalibrated Law under AdamW with Causal Validation [0.0]
  We derive a closed-form law T_grok - T_mem = (1 / 2 kappa_LL eta) log(V_mem / V_star), where V_t =theta_t||2 is the norm squared parameter, V_star is an architecture-dependent threshold, and kappa_LL absorbs the AdamW correction to the clean-SGD contraction rate 2 eta.<n> Calibrating (kappa_LL, V_star) on a single hyperlanguage cell predicts grokking delays on 26 held-out runs with MAPE 17.7% over a 41
  arXiv  Detail & Related papers  (2026-05-13T06:33:56Z)
• The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models [0.0]
  Empirically, three independent leaderboards (Open LLM v2, an extended 12-benchmark suite, LiveBench) all have d_eff in [2.86, 4.80] on their competitive frontier.<n>A submodular greedy algorithm with the Nemhauser (1 - 1/e) guarantee finds a stable core of 4 benchmarks.<n>A counterfactual validation across 12 internal benchmarks and 27 categories confirms that the eigenstructure predicts which evaluations are irreplaceable.
  arXiv  Detail & Related papers  (2026-04-15T08:56:58Z)
• Improved high-dimensional estimation with Langevin dynamics and stochastic weight averaging [54.172416732517156]
  Langevin dynamics can succeed with $n gtrsim d kstar/2 $ samples if one considers the average iterate, rather than the last.<n>We show that Langevin dynamics can succeed with $n gtrsim d kstar/2 $ samples if one considers the average iterate, rather than the last.
  arXiv  Detail & Related papers  (2026-03-06T08:23:24Z)
• Temporal Zoom Networks: Distance Regression and Continuous Depth for Efficient Action Localization [6.908972852063454]
  Temporal action localization requires both precise boundary detection and computational efficiency.<n>We address this through two complementary innovations: Boundary Distance Regression (BDR) and Adaptive Temporal Refinement (ATR)<n>On THUMOS14, our method achieves 56.5% mAP@0.7 with 151G FLOPs, using 36% fewer FLOPs than ActionFormer++ (55.7% mAP@0.7 at 235G)
  arXiv  Detail & Related papers  (2025-11-06T00:41:54Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Nearly Minimax Optimal Regret for Learning Linear Mixture Stochastic Shortest Path [80.60592344361073]
  We study the Shortest Path (SSP) problem with a linear mixture transition kernel. An agent repeatedly interacts with a environment and seeks to reach certain goal state while minimizing the cumulative cost. Existing works often assume a strictly positive lower bound of the iteration cost function or an upper bound of the expected length for the optimal policy.
  arXiv  Detail & Related papers  (2024-02-14T07:52:00Z)
• Vanishing Point Estimation in Uncalibrated Images with Prior Gravity Direction [82.72686460985297]
  We tackle the problem of estimating a Manhattan frame. We derive two new 2-line solvers, one of which does not suffer from singularities affecting existing solvers. We also design a new non-minimal method, running on an arbitrary number of lines, to boost the performance in local optimization.
  arXiv  Detail & Related papers  (2023-08-21T13:03:25Z)
• Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation [68.25872110275542]
  We propose an efficient inference procedure for non-autoregressive machine translation. It iteratively refines translation purely in the continuous space. We evaluate our approach on WMT'14 En-De, WMT'16 Ro-En and IWSLT'16 De-En.
  arXiv  Detail & Related papers  (2020-09-15T15:30:14Z)
• A Simple Convergence Proof of Adam and Adagrad [74.24716715922759]
  We show a proof of convergence between the Adam Adagrad and $O(d(N)/st)$ algorithms. Adam converges with the same convergence $O(d(N)/st)$ when used with the default parameters.
  arXiv  Detail & Related papers  (2020-03-05T01:56:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.