Spectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantization

• URL: http://arxiv.org/abs/2606.28432v1
• Date: Fri, 26 Jun 2026 02:10:53 GMT
• Title: Spectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantization
• Abstract summary: We study the spectral perturbation of the Fisher Information Matrix (FIM) of a parametric statistical model under two structured perturbations.<n>For the first, under an explicit local curvature-monotonicity hypothesis, we show departure from a reference manifold provably elevates the eigenvalue_max of the FIM.<n>Our principal result is a directional eigenvalue perturbation bound, via Weyl's inequality, showing the value of sigma_max under a quantization noise.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We study the spectral perturbation of the empirical Fisher Information Matrix (FIM) of a parametric statistical model under two structured perturbations: departure of the input from a reference (in-distribution) ensemble, and finite-precision (quantized) perturbation of the model's parameters. For the first, under an explicit local curvature-monotonicity hypothesis on the dominant eigenvalue lambda_max of the FIM, we show departure from a reference manifold provably elevates lambda_max relative to a calibration baseline (Proposition 3.2), and discuss why this hypothesis is required, since curvature need not increase monotonically under every perturbation. Our principal result is a directional eigenvalue perturbation bound, via Weyl's inequality, showing lambda_max under a quantization noise perturbation is lower bounded by its unperturbed value up to a third-order remainder, and, under a mild genericity condition, strictly exceeds it at leading order (Theorem 4.3). We give two tractable approximations to lambda_max -- one heuristic, one with a rigorous two-sided bound -- and a completeness result for a threshold-based partition of an augmented state space. These results motivate using sigma_t = lambda_max(F_t)/lambda_base as a runtime monitoring statistic for deployed language models: the quantization result offers a mechanism for an empirical observation of our own, where a calibration threshold for this statistic was approximately 244 times larger than a preliminary full-precision estimate on a 4-bit quantized model, a single measurement rather than a value derived in closed form. We report supporting measurements (twelve models, n=1,080 trajectories) broadly consistent with our predictions, discuss the scope and limitations of every result, and state as an open problem the closed-form prediction of the quantization inflation magnitude our bound does not supply.

Related papers

• Conditional Diffusion for Nonparametric Instrumental Variable Quantile Regression [18.254578337569264]
  This work proposes a two-stage estimator that combines conditional diffusion modeling with a kernel-parametriced conditional moment formulation.<n>We establish an excess-risk bound for the proposed estimator and derive end-to-end total variation guarantees for the conditional diffusion model.<n> Simulation studies and a real-data application demonstrate that the proposed method outperforms existing nonparametric IVQR approaches.
  arXiv  Detail & Related papers  (2026-08-08T16:01:24Z)
• Exact Closed-Form Quantum Correlations of Maximally Entangled Qudit States under Arbitrary Non-Markovian Pure Dephasing [0.0]
  We study the time evolution of entanglement and quantum discord for a pair of maximally entangled qudits of arbitrary dimension $d$ under non-Markovian pure dephasing.<n>Because this solution requires no Born--Markov, Lindblad, or rotating-wave approximation, every result reported here follows directly from the exact dynamics.
  arXiv  Detail & Related papers  (2026-07-22T03:05:59Z)
• Regularized Variational and Spectral Log-Density-Ratio Estimation in the Gaussian Location Model [0.554780083433538]
  We study ridge-regularized log-density-ratio estimation in the Gaussian location model.<n>The variational estimator is the empirical Kullback-Leibler (KL) log-normalized fit with a squared L2-penalty on its nonconstant coefficient.
  arXiv  Detail & Related papers  (2026-07-02T08:46:52Z)
• A Spectral Framework for Closed-Form Relative Density Estimation [0.554780083433538]
  We propose a closed-form spectral framework for relative log-density estimation in linearly parameterized probabilistic models.<n>We derive an explicit spectral formula based only on first- and second-order feature moments, yielding closed-form estimators of both divergences and log-density potentials for fixed features.
  arXiv  Detail & Related papers  (2026-05-11T14:51:17Z)
• Universality of General Spiked Tensor Models [9.454986540713655]
  We study the rank-one spiked tensor model in the high-dimensional regime.<n>We show that their high-dimensional spectral behavior and statistical limits are robust to non-Gaussian noise.
  arXiv  Detail & Related papers  (2026-02-04T11:59:30Z)
• Extracting quantum field theory dynamics from an approximate ground state [0.0]
  We develop a linear-programming method to extract information from static ground-state correlators in quantum field theory.<n>We test the method on the $1+1$-dimensional $4$ model, using a variational approximation to the vacuum.<n>The resulting mass gaps agree with renormalized Hamiltonian truncation and Borel-resummed perturbation theory.
  arXiv  Detail & Related papers  (2025-12-22T17:25:24Z)
• A General Class of Model-Free Dense Precision Matrix Estimators [0.0]
  We introduce prototype consistent model-free, dense precision matrix estimators that have broad application in economics.<n>We are able to: (i) obtain non-asymptotic bounds for precision matrix estimation errors; (ii) uncover the existence of an intrinsic signal-to-noise; and (iv) avoid exact population sparsity assumptions.<n>In addition to its desirable theoretical properties, a thorough empirical study of the S&P 500 index shows that a tuning parameter-free special case of our general estimator exhibits a doubly ascending Sharpe Ratio pattern.
  arXiv  Detail & Related papers  (2025-07-07T05:07:17Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Off-policy estimation of linear functionals: Non-asymptotic theory for semi-parametric efficiency [59.48096489854697]
  The problem of estimating a linear functional based on observational data is canonical in both the causal inference and bandit literatures. We prove non-asymptotic upper bounds on the mean-squared error of such procedures. We establish its instance-dependent optimality in finite samples via matching non-asymptotic local minimax lower bounds.
  arXiv  Detail & Related papers  (2022-09-26T23:50:55Z)
• Structural aspects of FRG in quantum tunnelling computations [68.8204255655161]
  We probe both the unidimensional quartic harmonic oscillator and the double well potential. Two partial differential equations for the potential V_k(varphi) and the wave function renormalization Z_k(varphi) are studied.
  arXiv  Detail & Related papers  (2022-06-14T15:23:25Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• A Random Matrix Perspective on Random Tensors [40.89521598604993]
  We study the spectra of random matrices arising from contractions of a given random tensor. Our technique yields a hitherto unknown characterization of the local maximum of the ML problem. Our approach is versatile and can be extended to other models, such as asymmetric, non-Gaussian and higher-order ones.
  arXiv  Detail & Related papers  (2021-08-02T10:42:22Z)
• Non-asymptotic Optimal Prediction Error for Growing-dimensional Partially Functional Linear Models [0.951828574518325]
  We show the rate-optimal upper and lower bounds of the prediction error. An exact upper bound for the excess prediction risk is shown in a non-asymptotic form. We derive the non-asymptotic minimax lower bound under the regularity assumption of the Kullback-Leibler divergence of the models.
  arXiv  Detail & Related papers  (2020-09-10T08:49:32Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions [41.7567932118769]
  Empirical Risk Minimization algorithms are widely used in a variety of estimation and prediction tasks. In this paper, we characterize for the first time the fundamental limits on the statistical accuracy of convex ERM for inference.
  arXiv  Detail & Related papers  (2020-06-16T04:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.