Curvature-Aligned Probing for Local Loss-Landscape Stabilization

• URL: http://arxiv.org/abs/2604.14870v1
• Date: Thu, 16 Apr 2026 11:01:48 GMT
• Title: Curvature-Aligned Probing for Local Loss-Landscape Stabilization
• Abstract summary: Local loss-landscape stabilization under sample growth is typically measured either pointwise or through isotropic averaging in the full parameter space.<n>We recast stabilization as an observational problem and introduce a unified family of criteria parameterized by an aggregation order and a probing distribution.<n>We derive scalable estimators based on Hessian-vector products, subspace Monte Carlo, and a closed-form-moment proxy.
• Score: 0.7090165638014331
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Local loss-landscape stabilization under sample growth is typically measured either pointwise or through isotropic averaging in the full parameter space. Despite practical value, both choices probe directions that contribute little to the dominant local deformation of strongly anisotropic neural landscapes. We recast stabilization as an observational problem and introduce a unified family of criteria parameterized by an aggregation order and a probing distribution; within this family we propose a curvature-aligned criterion $Δ_2^{(D)}$ that probes the loss increment field in the top-$D$ eigenspace of the empirical Hessian near a trained solution. Solely from a local quadratic model, we prove that $Δ_2^{(D)}$ preserves the $O(k^{-2})$ mean-squared rate of the full-space criterion while replacing ambient-dimension curvature dependence with dependence on the subspace dimension $D$; a corollary gives a closed-form spectral expression and a proposition identifies the top-$D$ eigenspace as extremal within the eigenspace-aligned family. We also derive scalable estimators based on Hessian-vector products, subspace Monte Carlo, and a closed-form Gaussian-moment proxy. On a decoder-only transformer, a curvature-aligned probe occupying a tiny fraction of parameter space already reproduces the full-space mean-squared signal to within numerical noise throughout the validated local regime, and the closed-form estimator is orders of magnitude faster than direct Monte Carlo after subspace construction.

Related papers

• Curvature-Independent Regret Bounds for Distributed Online Optimization on Hadamard Manifolds [9.035866986287205]
  This work addresses decentralized online convexian optimization on Hadamard.<n>To our knowledge these are the first curvature-independent regret guarantees on embeddings for predicted rates, with no degradation due to curvature.
  arXiv  Detail & Related papers  (2026-09-12T01:53:49Z)
• Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization [7.792020418343022]
  Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning.<n>We show that the two methods are locally consistent under parameter-space perturbations.<n>We also show that divergence-based regularization is defined through transformations of the input.
  arXiv  Detail & Related papers  (2026-09-08T19:01:05Z)
• Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks [3.980892205755342]
  We present an orientation-resolved quadratic master formula that predicts the random-slice residual from both the curvature spectrum and the reference-to-solution displacement profile.<n>We then introduce Random Mapping Networks (RaMaN), which instantiate the predicted latent dimension using structured Hadamard or seed-regenerated Gaussian maps.<n>RaMaN closely tracks measured transition locations and outperforms orientation-agnostic approximations when displacement direction matters.
  arXiv  Detail & Related papers  (2026-08-12T21:09:54Z)
• Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds [61.14405512940818]
  We show that Score-based generative models are trained in high-dimensional ambient spaces.<n>We use a ReLU implementation of nearest-projection coordinates via finite intrinsic anchors and Gauss Newton.
  arXiv  Detail & Related papers  (2026-05-15T10:20:05Z)
• Distributional Stability of Tangent-Linearized Gaussian Inference on Smooth Manifolds [3.632189127068905]
  We derive explicit non-asymptotic $W$ stability bounds for projection marginalization and surface-measure conditioning.<n>The bounds separate local second-order geometric distortion from nonlocal tail leakage.<n>Experiments validate the predicted calibration transition near $sqrt||_mathrmop/Rapprox 1/6$ and indicate that normal-direction uncertainty is the dominant failure mode when locality breaks.
  arXiv  Detail & Related papers  (2026-02-22T13:18:45Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Robust Tangent Space Estimation via Laplacian Eigenvector Gradient Orthogonalization [48.25304391127552]
  Estimating the tangent spaces of a data manifold is a fundamental problem in data analysis.<n>We propose a method, Laplacian Eigenvector Gradient Orthogonalization (LEGO), that utilizes the global structure of the data to guide local tangent space estimation.
  arXiv  Detail & Related papers  (2025-10-02T17:59:45Z)
• Dynamical local Fréchet curve regression in manifolds [0.0]
  We consider the prediction problem of the Earth's magnetic field from the time-varying geocentric latitude and longitude of NASA's MAGSAT spacecraft.<n>As motivating real data application, we consider the prediction problem of the Earth's magnetic field from the time-varying geocentric latitude and longitude of the satellite NASA's MAGSAT spacecraft.
  arXiv  Detail & Related papers  (2025-05-08T12:06:59Z)
• Density-Aware Noise Mechanisms for Differential Privacy on Riemannian Manifolds via Conformal Transformation [1.5632754424046598]
  We develop a density-aware differential privacy mechanism based on conformal transformations.<n>We demonstrate that our mechanism substantially improves the privacy-utility tradeoff in heterogeneous manifold settings.
  arXiv  Detail & Related papers  (2025-04-29T17:05:55Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Principal Component Analysis in Space Forms [7.822210329345704]
  We study Principal Component Analysis (PCA) in space forms. Finding the optimal low-dimensional affine subspace for given points in a space form amounts to dimensionality reduction. We propose proper cost functions that enjoy two properties: (1) their optimal affine subspace is the solution to an eigenequation, and (2) optimal affine subspaces of different dimensions form a nested set.
  arXiv  Detail & Related papers  (2023-01-06T23:48:37Z)
• Shape And Structure Preserving Differential Privacy [70.08490462870144]
  We show how the gradient of the squared distance function offers better control over sensitivity than the Laplace mechanism. We also show how using the gradient of the squared distance function offers better control over sensitivity than the Laplace mechanism.
  arXiv  Detail & Related papers  (2022-09-21T18:14:38Z)
• Optimal Scaling for Locally Balanced Proposals in Discrete Spaces [65.14092237705476]
  We show that efficiency of Metropolis-Hastings (M-H) algorithms in discrete spaces can be characterized by an acceptance rate that is independent of the target distribution. Knowledge of the optimal acceptance rate allows one to automatically tune the neighborhood size of a proposal distribution in a discrete space, directly analogous to step-size control in continuous spaces.
  arXiv  Detail & Related papers  (2022-09-16T22:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.