Certification of Machine Learning Models via Directional Sharpness

• URL: http://arxiv.org/abs/2606.25004v1
• Date: Tue, 23 Jun 2026 17:07:25 GMT
• Title: Certification of Machine Learning Models via Directional Sharpness
• Abstract summary: A model's quality is determined by its ability to generalize, i.e., to perform well on data beyond what it was trained on.<n>It is not possible to certify generalization directly, as it depends on unknown data and is not directly measurable.<n>We propose directional sharpness, a metric designed to efficiently and reliably indicate generalization despite potential training deviations.
• Score: 5.687933304893888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In machine learning, model certification has been identified as an important method for gaining assurance about a model's trustworthiness and quality. A model's quality is largely determined by its ability to generalize, i.e., to perform well on data beyond what it was trained on. It is not possible to certify generalization directly, however, as it depends on unknown data and is not directly measurable. Proxies such as test accuracy can be misleading when the training process is perturbed (intentionally or accidentally), and metrics such as sharpness -- which has an empirically supported link to generalization -- are computationally expensive and can also serve as unreliable signals when training deviates from a prescribed procedure. In this work, we propose directional sharpness, a metric designed to efficiently and reliably indicate generalization despite potential training deviations. We provide empirical and analytical evidence that directional sharpness (1) correlates more strongly with generalization than existing metrics and (2) identifies models with poor generalization more reliably than existing metrics. Furthermore, directional sharpness is efficiently computable in model auditing settings, where the verifier has access to training data, and via zero-knowledge proofs that certify quality without revealing training data.

Related papers

• Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget [25.03537230361785]
  We propose the first practical and general-purpose auditing framework for machine unlearning.<n>Our framework addresses the key limitations of existing methods by eliminating the need for retraining-from-scratch baselines.<n>We show that our framework is robust against fake unlearning attempts and generalizes well to large language models.
  arXiv  Detail & Related papers  (2026-06-15T01:56:09Z)
• When Models Know When They Do Not Know: Calibration, Cascading, and Cleaning [10.585100830578934]
  A promising approach is to use confidence, computed from the model's internal signals, to reflect its ignorance.<n>We propose a simple, effective, and universal training-free method that applies to both vision and language models.<n>Our results demonstrate that enabling models to recognize when they do not know is a practical step toward more efficient, reliable, and trustworthy AI.
  arXiv  Detail & Related papers  (2026-01-12T19:59:03Z)
• Lie Detector: Unified Backdoor Detection via Cross-Examination Framework [68.45399098884364]
  We propose a unified backdoor detection framework in the semi-honest setting.<n>Our method achieves superior detection performance, improving accuracy by 5.4%, 1.6%, and 11.9% over SoTA baselines.<n> Notably, it is the first to effectively detect backdoors in multimodal large language models.
  arXiv  Detail & Related papers  (2025-03-21T06:12:06Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Adaptive Certified Training: Towards Better Accuracy-Robustness Tradeoffs [17.46692880231195]
  We propose a novel certified training method based on a key insight that training with adaptive certified radii helps to improve the accuracy and robustness of the model. We demonstrate the effectiveness of the proposed method on MNIST, CIFAR-10, and TinyImageNet datasets.
  arXiv  Detail & Related papers  (2023-07-24T18:59:46Z)
• Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [28.522673618527417]
  We propose a novel Bayesian meta-model to augment pre-trained models with better uncertainty quantification abilities. Our proposed method requires no additional training data and is flexible enough to quantify different uncertainties. We demonstrate our proposed meta-model approach's flexibility and superior empirical performance on these applications.
  arXiv  Detail & Related papers  (2022-12-14T17:34:11Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification [58.03725169462616]
  We show theoretically that over-parametrization is not the only reason for over-confidence. We prove that logistic regression is inherently over-confident, in the realizable, under-parametrized setting. Perhaps surprisingly, we also show that over-confidence is not always the case.
  arXiv  Detail & Related papers  (2021-02-15T21:38:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.