A Function Centric Perspective On Flat and Sharp Minima

• URL: http://arxiv.org/abs/2510.12451v1
• Date: Tue, 14 Oct 2025 12:33:14 GMT
• Title: A Function Centric Perspective On Flat and Sharp Minima
• Authors: Israel Mason-Williams, Gabryel Mason-Williams, Helen Yannakoudakis,
• Abstract summary: Flat minima are widely believed to correlate with improved generalisation in deep neural networks.<n>This paper proposes that sharpness is better understood as a function-dependent property rather than a reliable indicator of poor generalisation.
• Score: 4.908739793053431
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flat minima are widely believed to correlate with improved generalisation in deep neural networks. However, this connection has proven more nuanced in recent studies, with both theoretical counterexamples and empirical exceptions emerging in the literature. In this paper, we revisit the role of sharpness in model performance, proposing that sharpness is better understood as a function-dependent property rather than a reliable indicator of poor generalisation. We conduct extensive empirical studies, from single-objective optimisation to modern image classification tasks, showing that sharper minima often emerge when models are regularised (e.g., via SAM, weight decay, or data augmentation), and that these sharp minima can coincide with better generalisation, calibration, robustness, and functional consistency. Across a range of models and datasets, we find that baselines without regularisation tend to converge to flatter minima yet often perform worse across all safety metrics. Our findings demonstrate that function complexity, rather than flatness alone, governs the geometry of solutions, and that sharper minima can reflect more appropriate inductive biases (especially under regularisation), calling for a function-centric reappraisal of loss landscape geometry.

Related papers

• Flatness After All? [6.977444416330261]
  We argue that generalization could be assessed by measuring flatness using a soft rank measure of the Hessian.<n>For non-calibrated models, we connect a soft rank based flatness measure to the well-known Takeuchi Information Criterion.
  arXiv  Detail & Related papers  (2025-06-21T20:33:36Z)
• Simplicity Bias via Global Convergence of Sharpness Minimization [43.658859631741024]
  We show that label noise SGD always minimizes the sharpness on the manifold of models with zero loss for two-layer networks. We also find a novel property of the trace of Hessian of the loss at approximate stationary points on the manifold of zero loss.
  arXiv  Detail & Related papers  (2024-10-21T18:10:37Z)
• A Universal Class of Sharpness-Aware Minimization Algorithms [57.29207151446387]
  We introduce a new class of sharpness measures, leading to new sharpness-aware objective functions. We prove that these measures are textitly expressive, allowing any function of the training loss Hessian matrix to be represented by appropriate hyper and determinants.
  arXiv  Detail & Related papers  (2024-06-06T01:52:09Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• FAM: Relative Flatness Aware Minimization [5.132856559837775]
  optimizing for flatness has been proposed as early as 1994 by Hochreiter and Schmidthuber. Recent theoretical work suggests that a particular relative flatness measure can be connected to generalization. We derive a regularizer based on this relative flatness that is easy to compute, fast, efficient, and works with arbitrary loss functions.
  arXiv  Detail & Related papers  (2023-07-05T14:48:24Z)
• Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning [70.52689048213398]
  This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features. We find new and interesting properties that do not exist in single-task linear regression. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
  arXiv  Detail & Related papers  (2023-04-09T20:36:13Z)
• Predicting and Enhancing the Fairness of DNNs with the Curvature of Perceptual Manifolds [44.79535333220044]
  Recent studies have shown that tail classes are not always hard to learn, and model bias has been observed on sample-balanced datasets.<n>In this work, we first establish a geometric perspective for analyzing model fairness and then systematically propose a series of geometric measurements.
  arXiv  Detail & Related papers  (2023-03-22T04:49:23Z)
• Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization [33.50116027503244]
  We show that the zeroth-order flatness can be insufficient to discriminate minima with low gradient error. We also present a novel training procedure named Gradient norm Aware Minimization (GAM) to seek minima with uniformly small curvature across all directions.
  arXiv  Detail & Related papers  (2023-03-03T16:58:53Z)
• A Modern Look at the Relationship between Sharpness and Generalization [64.03012884804458]
  Sharpness of minima is promising quantity that can correlate with generalization in deep networks. Sharpness is not invariant under reparametrizations of neural networks. We show that sharpness does not correlate well with generalization.
  arXiv  Detail & Related papers  (2023-02-14T12:38:12Z)
• Improving Generalization in Federated Learning by Seeking Flat Minima [23.937135834522145]
  Models trained in federated settings often suffer from degraded performances and fail at generalizing. In this work, we investigate such behavior through the lens of geometry of the loss and Hessian eigenspectrum. Motivated by prior studies connecting the sharpness of the loss surface and the generalization gap, we show that i) training clients locally with Sharpness-Aware Minimization (SAM) or its adaptive version (ASAM) on the server-side can substantially improve generalization.
  arXiv  Detail & Related papers  (2022-03-22T16:01:04Z)
• Generalization of Neural Combinatorial Solvers Through the Lens of Adversarial Robustness [68.97830259849086]
  Most datasets only capture a simpler subproblem and likely suffer from spurious features. We study adversarial robustness - a local generalization property - to reveal hard, model-specific instances and spurious features. Unlike in other applications, where perturbation models are designed around subjective notions of imperceptibility, our perturbation models are efficient and sound. Surprisingly, with such perturbations, a sufficiently expressive neural solver does not suffer from the limitations of the accuracy-robustness trade-off common in supervised learning.
  arXiv  Detail & Related papers  (2021-10-21T07:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.