PostHoc FREE Calibrating on Kolmogorov Arnold Networks

• URL: http://arxiv.org/abs/2503.01195v1
• Date: Mon, 03 Mar 2025 05:42:49 GMT
• Title: PostHoc FREE Calibrating on Kolmogorov Arnold Networks
• Authors: Wenhao Liang, Wei Emma Zhang, Lin Yue, Miao Xu, Olaf Maennel, Weitong Chen,
• Abstract summary: Kolmogorov Arnold Networks (KANs) are neural architectures inspired by the Kolmogorov Arnold representation theorem.<n>KANs can capture complex nonlinearities beyond those modeled by standard MultiLayer Perceptrons (MLPs)<n>KANs frequently exhibit miscalibrated confidence estimates manifesting as overconfidence in dense data regions and underconfidence in sparse areas.
• Score: 16.957071012748454
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Kolmogorov Arnold Networks (KANs) are neural architectures inspired by the Kolmogorov Arnold representation theorem that leverage B Spline parameterizations for flexible, locally adaptive function approximation. Although KANs can capture complex nonlinearities beyond those modeled by standard MultiLayer Perceptrons (MLPs), they frequently exhibit miscalibrated confidence estimates manifesting as overconfidence in dense data regions and underconfidence in sparse areas. In this work, we systematically examine the impact of four critical hyperparameters including Layer Width, Grid Order, Shortcut Function, and Grid Range on the calibration of KANs. Furthermore, we introduce a novel TemperatureScaled Loss (TSL) that integrates a temperature parameter directly into the training objective, dynamically adjusting the predictive distribution during learning. Both theoretical analysis and extensive empirical evaluations on standard benchmarks demonstrate that TSL significantly reduces calibration errors, thereby improving the reliability of probabilistic predictions. Overall, our study provides actionable insights into the design of spline based neural networks and establishes TSL as a robust loss solution for enhancing calibration.

Related papers

• Imitation Learning of MPC with Neural Networks: Error Guarantees and Sparsification [5.260346080244568]
  We present a framework for bounding the approximation error in imitation model predictive controllers utilizing neural networks.<n>We discuss how this method can be used to design a stable neural network controller with performance guarantees.
  arXiv  Detail & Related papers  (2025-01-07T10:18:37Z)
• Calibrating Deep Neural Network using Euclidean Distance [5.675312975435121]
  In machine learning, Focal Loss is commonly used to reduce misclassification rates by emphasizing hard-to-classify samples. High calibration error indicates a misalignment between predicted probabilities and actual outcomes, affecting model reliability. This research introduces a novel loss function called Focal Loss (FCL), designed to improve probability calibration while retaining the advantages of Focal Loss in handling difficult samples.
  arXiv  Detail & Related papers  (2024-10-23T23:06:50Z)
• Positional Encoder Graph Quantile Neural Networks for Geographic Data [4.277516034244117]
  We introduce the Positional Graph Quantile Neural Network (PE-GQNN), a novel method that integrates PE-GNNs, Quantile Neural Networks, and recalibration techniques in a fully nonparametric framework. Experiments on benchmark datasets demonstrate that PE-GQNN significantly outperforms existing state-of-the-art methods in both predictive accuracy and uncertainty quantification.
  arXiv  Detail & Related papers  (2024-09-27T16:02:12Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• On the Calibration and Uncertainty with P\'{o}lya-Gamma Augmentation for Dialog Retrieval Models [30.519215651368683]
  dialog response retrieval models output a single score for a response on how relevant it is to a given question. Bad calibration of deep neural network results in various uncertainty for the single score such that the unreliable predictions always misinform user decisions. We present an efficient calibration and uncertainty estimation framework PG-DRR for dialog response retrieval models.
  arXiv  Detail & Related papers  (2023-03-15T13:26:25Z)
• Differentially private training of neural networks with Langevin dynamics forcalibrated predictive uncertainty [58.730520380312676]
  We show that differentially private gradient descent (DP-SGD) can yield poorly calibrated, overconfident deep learning models. This represents a serious issue for safety-critical applications, e.g. in medical diagnosis.
  arXiv  Detail & Related papers  (2021-07-09T08:14:45Z)
• Training Sparse Neural Network by Constraining Synaptic Weight on Unit Lp Sphere [2.429910016019183]
  constraining the synaptic weights on unit Lp-sphere enables the flexibly control of the sparsity with p. Our approach is validated by experiments on benchmark datasets covering a wide range of domains.
  arXiv  Detail & Related papers  (2021-03-30T01:02:31Z)
• Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration [57.568461777747515]
  We introduce a novel calibration method, Parametrized Temperature Scaling (PTS) We demonstrate that the performance of accuracy-preserving state-of-the-art post-hoc calibrators is limited by their intrinsic expressive power. We show with extensive experiments that our novel accuracy-preserving approach consistently outperforms existing algorithms across a large number of model architectures, datasets and metrics.
  arXiv  Detail & Related papers  (2021-02-24T10:18:30Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime [50.510421854168065]
  We show that the averaged gradient descent can achieve the minimax optimal convergence rate. We show that the target function specified by the NTK of a ReLU network can be learned at the optimal convergence rate.
  arXiv  Detail & Related papers  (2020-06-22T14:31:37Z)
• Calibrating Deep Neural Networks using Focal Loss [77.92765139898906]
  Miscalibration is a mismatch between a model's confidence and its correctness. We show that focal loss allows us to learn models that are already very well calibrated. We show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases.
  arXiv  Detail & Related papers  (2020-02-21T17:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.