ACLS: Adaptive and Conditional Label Smoothing for Network Calibration

• URL: http://arxiv.org/abs/2308.11911v2
• Date: Thu, 24 Aug 2023 06:35:22 GMT
• Title: ACLS: Adaptive and Conditional Label Smoothing for Network Calibration
• Authors: Hyekang Park, Jongyoun Noh, Youngmin Oh, Donghyeon Baek, Bumsub Ham
• Abstract summary: Many approaches to network calibration adopt a regularization-based method that exploits a regularization term to smooth the miscalibrated confidences. We present in this paper an in-depth analysis of existing regularization-based methods, providing a better understanding on how they affect to network calibration. We introduce a novel loss function, dubbed ACLS, that unifies the merits of existing regularization methods, while avoiding the limitations.
• Score: 30.80635918457243
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the problem of network calibration adjusting miscalibrated confidences of deep neural networks. Many approaches to network calibration adopt a regularization-based method that exploits a regularization term to smooth the miscalibrated confidences. Although these approaches have shown the effectiveness on calibrating the networks, there is still a lack of understanding on the underlying principles of regularization in terms of network calibration. We present in this paper an in-depth analysis of existing regularization-based methods, providing a better understanding on how they affect to network calibration. Specifically, we have observed that 1) the regularization-based methods can be interpreted as variants of label smoothing, and 2) they do not always behave desirably. Based on the analysis, we introduce a novel loss function, dubbed ACLS, that unifies the merits of existing regularization methods, while avoiding the limitations. We show extensive experimental results for image classification and semantic segmentation on standard benchmarks, including CIFAR10, Tiny-ImageNet, ImageNet, and PASCAL VOC, demonstrating the effectiveness of our loss function.

Related papers

• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• An Uncertainty-Aware Pseudo-Label Selection Framework using Regularized Conformal Prediction [0.0]
  Pseudo-labeling (PL) is a general and domain-agnostic SSL approach. PL underperforms due to the erroneous high-confidence predictions from poorly calibrated models. This paper proposes an uncertainty-aware pseudo-label selection framework.
  arXiv  Detail & Related papers  (2023-08-30T17:13:30Z)
• Variational Inference: Posterior Threshold Improves Network Clustering Accuracy in Sparse Regimes [2.5782420501870296]
  This paper proposes a simple way to improve the variational inference method by hard thresholding the posterior of the community assignment after each iteration. We show that the proposed method converges and can accurately recover the true community labels, even when the average node degree of the network is bounded.
  arXiv  Detail & Related papers  (2023-01-12T00:24:54Z)
• Calibrating Segmentation Networks with Margin-based Label Smoothing [19.669173092632]
  We provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. These losses could be viewed as approximations of a linear penalty imposing equality constraints on logit distances. We propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.
  arXiv  Detail & Related papers  (2022-09-09T20:21:03Z)
• Distribution Mismatch Correction for Improved Robustness in Deep Neural Networks [86.42889611784855]
  normalization methods increase the vulnerability with respect to noise and input corruptions. We propose an unsupervised non-parametric distribution correction method that adapts the activation distribution of each layer. In our experiments, we empirically show that the proposed method effectively reduces the impact of intense image corruptions.
  arXiv  Detail & Related papers  (2021-10-05T11:36:25Z)
• Sparsity Aware Normalization for GANs [32.76828505875087]
  Generative adversarial networks (GANs) are known to benefit from regularization or normalization of their critic (discriminator) network during training. In this paper, we analyze the popular spectral normalization scheme, find a significant drawback and introduce sparsity aware normalization (SAN), a new alternative approach for stabilizing GAN training.
  arXiv  Detail & Related papers  (2021-03-03T15:05:18Z)
• Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification [17.941506832422192]
  Empirically, neural networks are often miscalibrated and overconfident in their predictions. We propose a novel calibration approach that maintains the overall classification accuracy while significantly improving model calibration.
  arXiv  Detail & Related papers  (2020-09-09T01:25:53Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• Post-hoc Calibration of Neural Networks by g-Layers [51.42640515410253]
  In recent years, there is a surge of research on neural network calibration. It is known that minimizing Negative Log-Likelihood (NLL) will lead to a calibrated network on the training set if the global optimum is attained. We prove that even though the base network ($f$) does not lead to the global optimum of NLL, by adding additional layers ($g$) and minimizing NLL by optimizing the parameters of $g$ one can obtain a calibrated network.
  arXiv  Detail & Related papers  (2020-06-23T07:55:10Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)
• Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks [54.23874144090228]
  A common approach is to learn a post-hoc calibration function that transforms the output of the original network into calibrated confidence scores. Previous post-hoc calibration techniques work only with simple calibration functions. We propose a new neural network architecture that represents a class of intra order-preserving functions.
  arXiv  Detail & Related papers  (2020-03-15T12:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.