Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training

• URL: http://arxiv.org/abs/2605.25674v1
• Date: Mon, 25 May 2026 10:24:32 GMT
• Title: Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
• Abstract summary: We present a parameter estimator of the trace of the diagonal blocks of the Hessian matrix of the empirical risk of a neural network.<n>The procedure combines the Hutchinson trace estimator with a single Hessian-vector product over the whole parameter vector.<n>We show that correctness under weight sharing requires the layer-wise Hessian to be assembled before the second differentiation.
• Score: 45.88028371034407
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The loss and the norm of its gradient separate the healthy and the pathological regimes of neural-network training only weakly, whilst the curvature of the empirical risk differs qualitatively between them but is inaccessible explicitly at parameter counts $P\sim 10^{6}-10^{8}$. We present a stochastic estimator of the trace of the diagonal blocks of the Hessian matrix of the empirical risk of a neural network. The procedure combines the Hutchinson stochastic trace estimator with a single Hessian-vector product over the whole parameter vector and recovers unbiased estimates of every per-layer trace in one backward pass through the computational graph. We show that correctness under weight sharing requires the layer-wise Hessian to be assembled before the second differentiation: unrolling shared weights into independent coordinates introduces a systematic bias whose sign and magnitude are governed by the cross-instance blocks of the unrolled Hessian. A closed-form expression for the variance of the estimator at a fixed Hessian is derived, together with a decomposition of the total variance under the mini-batch sampling distribution. This decomposition yields a critical probe count $K^{\star}$ that balances the two sources of randomness and supports the practical recommendation $K\in[5,10]$ in the on-line monitoring regime. The estimator is applied to the detection of the label-memorisation regime of ResNet-18, ResNet-34, and VGG-11 on CIFAR-10 and CIFAR-100, where a calibrated cumulative-sum decision rule attains an empirical detection power of $179/180$ at a false-alarm rate of $16/120$.

Related papers

• Finite-Probe Total-Variation Certificates for Finite-Basis Drifting Models [1.2072720711193232]
  We describe a frozen measurement system for integrable antisymmetric interactions and absolutely continuous laws in a declared finite density basis.<n>For Gaussian-RBF interactions, a global envelope supports distribution-free and empirical-Bernstein radii without truncation.<n>We characterize random-probe observability by a population Gram matrix, identify rank and symmetry degenera, and prove large-bandwidth collapse toward mean matching.
  arXiv  Detail & Related papers  (2026-08-03T00:00:40Z)
• Active Quantum Kernel Acquisition for Gaussian Process Regression [30.032230630492155]
  Recent work on quantum kernel classification has shown that allocating shots non-uniformly across kernel entries can reduce the shot budget required to reach a target accuracy.<n>We extend this idea to Gaussian process (GP) regression, a setting whose downstream quantities (full-spectrum posterior variance, log-determinant, marginal likelihood) couple to kernel error more tightly than the sign-only outputs of classification.<n>We derive three closed-form pair-level sensitivities predictive coupling $|_i_j|$, leave-one-out residual, and marginal-likelihood gradient and plug them into a Neyman-style minimum
  arXiv  Detail & Related papers  (2026-06-27T09:40:20Z)
• Structure-Aware Variance Reduction for Unbiased Randomized Hamiltonian Simulation [0.0]
  Hamiltonian simulation methods are often governed by a trade-off between systematic bias and sampling overhead.<n>We show how classical variance-reduction techniques can be applied to such methods without changing their mean channel.
  arXiv  Detail & Related papers  (2026-06-22T16:17:55Z)
• Kernel-based potential mean-field games with unbiased random Fourier $U$-statistics [0.0]
  We study the subclass of potential mean-field games in which the running interaction cost and the terminal target cost are both expressed through reproducing- kernel maximum mean discrepancy penalties.<n>Both costs are estimated from finite-sample empirical distributions using a random Fourier U-statistic representation that is unbiased and has linear cost in the batch size.
  arXiv  Detail & Related papers  (2026-05-28T05:15:25Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements [66.94250413799232]
  We study mean estimation of a random vector $X$ in a distributed parameter-server-worker setup.<n>Key challenges are adversarial measurements and asynchrony.<n>Results provide a unified finite-time characterization of robustness, identifiability, and statistical efficiency in distributed linear estimation with adversarial workers.
  arXiv  Detail & Related papers  (2026-04-07T11:45:55Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Fundamental Limits of Community Detection in Contextual Multi-Layer Stochastic Block Models [4.312207944236305]
  We consider the problem of community detection from the joint observation of a high-dimensional covariate matrix and $L$ sparse networks.<n>In the sparse regime where the networks have constant average degree and the number of features $p$ grows proportionally with $n$, we derive a sharp threshold under which detecting and estimating the subject labels is possible.<n>Our results show that there is no statistical-computational gap in this setting.
  arXiv  Detail & Related papers  (2026-02-09T00:27:54Z)
• Analysis of Hessian Scaling for Local and Global Costs in Variational Quantum Algorithm [0.42970700836450487]
  We quantify the entrywise resolvability of the Hessian for Variational Quantum Algorithms.<n>We show two distinct scaling regimes that govern the sample complexity required to resolve Hessian entries against shot noise.
  arXiv  Detail & Related papers  (2026-01-31T15:49:23Z)
• Benign Overfitting in Out-of-Distribution Generalization of Linear Models [19.203753135860016]
  We take an initial step towards understanding benign overfitting in the Out-of-Distribution (OOD) regime.<n>We provide non-asymptotic guarantees proving that benign overfitting occurs in standard ridge regression.<n>We also present theoretical results for a more general family of target covariance matrix.
  arXiv  Detail & Related papers  (2024-12-19T02:47:39Z)
• Langevin dynamics for high-dimensional optimization: the case of multi-spiked tensor PCA [8.435118770300999]
  We show that the sample complexity required for recovering the spike associated with the largest SNR matches the well-known algorithmic threshold for the single-spike case.<n>As a key step, we provide a detailed characterization of the spikes and interactions that capture the high-dimensional trajectory dynamics.
  arXiv  Detail & Related papers  (2024-08-12T12:09:25Z)
• Estimation of Non-Crossing Quantile Regression Process with Deep ReQU Neural Networks [5.5272015676880795]
  We propose a penalized nonparametric approach to estimating the quantile regression process (QRP) in a nonseparable model using quadratic unit (ReQU) activated deep neural networks. We establish the non-asymptotic excess risk bounds for the estimated QRP and derive the mean integrated squared error for the estimated QRP under mild smoothness and regularity conditions.
  arXiv  Detail & Related papers  (2022-07-21T12:26:45Z)
• Unrolling Particles: Unsupervised Learning of Sampling Distributions [102.72972137287728]
  Particle filtering is used to compute good nonlinear estimates of complex systems. We show in simulations that the resulting particle filter yields good estimates in a wide range of scenarios.
  arXiv  Detail & Related papers  (2021-10-06T16:58:34Z)
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification [54.22421582955454]
  We provide the first result of the optimal minimax guarantees for the excess risk for adversarially robust classification. Results are stated in terms of the Adversarial Signal-to-Noise Ratio (AdvSNR), which generalizes a similar notion for standard linear classification to the adversarial setting.
  arXiv  Detail & Related papers  (2020-06-29T21:06:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.