Exploiting weight-space symmetries for approximating curvature

• URL: http://arxiv.org/abs/2606.00442v1
• Date: Sat, 30 May 2026 00:17:36 GMT
• Title: Exploiting weight-space symmetries for approximating curvature
• Abstract summary: We develop a framework for approximating a loss function's curvature at the scale of modern deep networks.<n>Our framework might find applications in other machine learning problems such as uncertainty estimation, continual learning, compression/pruning, and more.
• Score: 39.60881703969561
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many machine learning techniques rely on approximating a loss function's curvature, but this is notoriously hard to do at the scale of modern deep networks. Surprisingly, no previous work has exploited the curvature constraints that arise from well known weight-space symmetries in loss landscapes. By analytically averaging over group actions that leave the loss invariant, we construct structured Hessian approximations from single gradients that can be tractably estimated, stored, and inverted. The choice of user-specified symmetry group directly governs the trade-off between approximation accuracy and computational cost. Moreover, our framework provides a unifying theoretical lens for viewing existing methods; in particular, a specific choice of symmetry group recovers Shampoo/Muon-like curvature estimates. We validate our method on a range of network architectures, and deploy it to second-order optimization benchmarks, including a small language model. Our curvature estimation framework might find applications in other machine learning problems such as uncertainty estimation, continual learning, compression/pruning, training data attribution, and more.

Related papers

• Semi-Dense Matching Uncertainty Is Not Just Local Confidence [0.0]
  We propose a lightweight, post-hoc overall uncertainty estimation framework that introduces a two-component Laplace mixture model with only 9 learnable parameters.<n>The objective is to explicitly capture both the sharp local refinement noise and the broader tail of coarse-assignment failures.<n>Our method consistently improves downstream geometric accuracy across various pretrained-only matchers and robust estimators with minimal computational overhead.
  arXiv  Detail & Related papers  (2026-08-09T13:01:20Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• Model Merging on Loss Landscape: A Geometry Perspective [2.2624434825206543]
  EpiMer is a framework that casts model merging as solving the Fréchet mean on a Riemannian manifold.<n>Our theoretical analysis decomposes the merging error bound into the subspace Fréchet variance and the residual energy.<n>Merging fine-tuned CLIP-ViT models on eight image classification tasks, Epistemic Merging strictly outperforms the baselines on all three CLIP-ViT backbones at matched rank.
  arXiv  Detail & Related papers  (2026-05-26T08:33:30Z)
• Practical Global and Local Bounds in Gaussian Process Regression via Chaining [4.500208956289746]
  We propose a chaining-based framework for estimating upper and lower bounds on the expected extreme values over unseen data.<n>We also develop a novel method for local uncertainty quantification at specified inputs.
  arXiv  Detail & Related papers  (2025-11-12T09:30:01Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectives [0.0]
  We show that neural network training reliably fails when relying on exact curvature information.<n>The failure modes provide insight both into the geometry of nonlinear discretizations as well as the distribution of stationary points in the loss landscape.
  arXiv  Detail & Related papers  (2025-10-13T22:29:52Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate.<n>We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Distributed Sketching for Randomized Optimization: Exact Characterization, Concentration and Lower Bounds [54.51566432934556]
  We consider distributed optimization methods for problems where forming the Hessian is computationally challenging. We leverage randomized sketches for reducing the problem dimensions as well as preserving privacy and improving straggler resilience in asynchronous distributed systems.
  arXiv  Detail & Related papers  (2022-03-18T05:49:13Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.