Bypassing Minimization Bias: A Shift-Invariant Variance Estimator for Off-Equilibrium Local Learning Coefficients

• URL: http://arxiv.org/abs/2606.22389v1
• Date: Sun, 21 Jun 2026 08:32:34 GMT
• Title: Bypassing Minimization Bias: A Shift-Invariant Variance Estimator for Off-Equilibrium Local Learning Coefficients
• Abstract summary: Shift-Invariant Variance Estimator (SIVE)<n>SIVE is a variance-based local LLC probe that structurally eliminates the unknown additive baseline through the variance operator.<n>Applying to deep neural networks, SIVE provides a robust, localized online diagnostic for tracking structural phase transitions throughout training.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Singular Learning Theory leverages the Local Learning Coefficient (LLC) to quantify the geometry of neural network loss landscapes. However, mean-energy LLC estimators depend explicitly on an additive loss baseline, typically an estimate of the local minimum. During transient, off-equilibrium training phases, this minimum is unknown; substituting it with the lowest noisy mini-batch loss induces a systematic minimization bias that distorts the geometric measurement. In this paper, we propose the Shift-Invariant Variance Estimator (SIVE), a variance-based local LLC probe that structurally eliminates the unknown additive baseline through the variance operator. Combining this shift-invariant observable with an explicit correction derived from the Law of Total Variance, SIVE separates geometric loss fluctuations from mini-batch evaluation noise. Controlled experiments on analytically tractable toy models show that SIVE recovers the expected finite-temperature geometric signal in regimes where anchored mean estimators fail. Applied to deep neural networks, SIVE provides a robust, localized online diagnostic for tracking structural phase transitions throughout training.

Related papers

• Scalable Lindblad Noise Learning via Stochastic Tensor-Network Simulation [48.887627688666974]
  Learning dissipation rates in large-scale open quantum systems is a major obstacle for near-term quantum technologies.<n>Here, we propose a scalable noise-learning framework for Lindblad dissipation rates that combines a simulation method, the Jump Method (TJM)<n>We demonstrate the approach on two noise models in the Ising model, in which independent dissipation rates are learned for each site up to $N_mathrmsite=16$.<n>This combination of scalable numerics and rigorous theoretical guarantees positions TJM-based noise learning as a practical foundation for characterizing dissipation in large quantum devices.
  arXiv  Detail & Related papers  (2026-08-25T15:07:11Z)
• Probabilistic Physics-Informed Neural Networks for Estimating Heterogeneous Elastic Properties from Low-Resolution and Noisy Displacement Data [4.791233143264229]
  Estimating elastic properties from low-resolution displacement measurements is a severely ill-posed inverse elasticity problem.<n>Existing inverse methods often rely on high-fidelity observations and manually prespecified loss weights.<n>We propose a Probabilistic Inverse Elasticity Physics-Informed Neural Network (PIE-PINN) framework for robust estimation.
  arXiv  Detail & Related papers  (2026-07-16T04:43:39Z)
• Unified Geometry-Guided ML-FTLE for Tracking Transient Chaos from Scalar Time Series [0.0]
  We propose a geometry-guided machine learning framework that unifies predictive trajectory divergence with macroscopic attractor morphology to track abrupt regime shifts.<n>This equation-free framework provides a highly accurate, noise-resilient diagnostic for monitoring structural transitions in complex non-stationary systems.
  arXiv  Detail & Related papers  (2026-06-05T15:22:57Z)
• Variationally correct operator learning: Reduced basis neural operator with a posteriori error estimation [3.8135482236014133]
  Minimizing PDE-residual losses is a common strategy to promote physical consistency in neural operators.<n>This work develops a variationally correct operator learning framework by constructing first-order system least-squares (FOSLS) objectives.<n>We provide a rigorous convergence analysis that bounds the total error by the sum of finite element discretization bias, reduced basis truncation error, neural network approximation error, and statistical estimation errors.
  arXiv  Detail & Related papers  (2025-12-24T18:37:59Z)
• Precise Bayesian Neural Networks [0.0]
  We develop a lightweight, implementation-ready variational unit that fits modern normalized architectures and improves calibration without sacrificing accuracy.<n>In short, by aligning the variational posterior with the network's intrinsic geometry, BNNs can be simultaneously principled, practical, and precise.
  arXiv  Detail & Related papers  (2025-06-24T15:42:00Z)
• Modes of Sequence Models and Learning Coefficients [0.6906005491572401]
  We develop a geometric account of sequence modelling that links patterns in the data to measurable properties of the loss landscape in transformer networks.<n>We show theoretically that Local Learning Coefficient estimates are insensitive to modes below a data-dependent threshold.<n>This insight clarifies why reliable LLC estimates can be obtained even when a network parameter is not a strict minimiser of the population loss.
  arXiv  Detail & Related papers  (2025-04-25T03:38:10Z)
• Leveraging Nested MLMC for Sequential Neural Posterior Estimation with Intractable Likelihoods [0.38233569758620045]
  Methods aim to learn the posterior from adaptively proposed simulations using neural network-based conditional density estimators.<n>The automatic posterior transformation (APT) method proposed by Greenberg et al. performs well and scales to high-level runtime data.<n>In this paper, we reformulate APT as a nested estimation problem.<n>We construct several multi- Monte Carlo (MLMC) estimators for the loss function and its gradients to accommodate different scenarios.
  arXiv  Detail & Related papers  (2024-01-30T06:29:41Z)
• Learning Discretized Neural Networks under Ricci Flow [48.47315844022283]
  We study Discretized Neural Networks (DNNs) composed of low-precision weights and activations.<n>DNNs suffer from either infinite or zero gradients due to the non-differentiable discrete function during training.
  arXiv  Detail & Related papers  (2023-02-07T10:51:53Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• On Convergence of Training Loss Without Reaching Stationary Points [62.41370821014218]
  We show that Neural Network weight variables do not converge to stationary points where the gradient the loss function vanishes. We propose a new perspective based on ergodic theory dynamical systems.
  arXiv  Detail & Related papers  (2021-10-12T18:12:23Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment [52.02794488304448]
  We propose a new distribution alignment method based on a log-likelihood ratio statistic and normalizing flows. We experimentally verify that minimizing the resulting objective results in domain alignment that preserves the local structure of input domains.
  arXiv  Detail & Related papers  (2020-03-26T22:10:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.