Enhance GNNs with Reliable Confidence Estimation via Adversarial Calibration Learning

• URL: http://arxiv.org/abs/2503.18235v1
• Date: Sun, 23 Mar 2025 23:04:41 GMT
• Title: Enhance GNNs with Reliable Confidence Estimation via Adversarial Calibration Learning
• Authors: Yilong Wang, Jiahao Zhang, Tianxiang Zhao, Suhang Wang,
• Abstract summary: Despite their impressive predictive performance, GNNs often exhibit poor confidence calibration.<n>This issue raises concerns about their reliability in high-stakes domains such as fraud detection, and risk assessment.<n>We propose a novel AdvCali framework that adaptively enhances calibration across different node groups.
• Score: 30.450482094196243
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite their impressive predictive performance, GNNs often exhibit poor confidence calibration, i.e., their predicted confidence scores do not accurately reflect true correctness likelihood. This issue raises concerns about their reliability in high-stakes domains such as fraud detection, and risk assessment, where well-calibrated predictions are essential for decision-making. To ensure trustworthy predictions, several GNN calibration methods are proposed. Though they can improve global calibration, our experiments reveal that they often fail to generalize across different node groups, leading to inaccurate confidence in node groups with different degree levels, classes, and local structures. In certain cases, they even degrade calibration compared to the original uncalibrated GNN. To address this challenge, we propose a novel AdvCali framework that adaptively enhances calibration across different node groups. Our method leverages adversarial training to automatically identify mis-calibrated node groups and applies a differentiable Group Expected Calibration Error (ECE) loss term to refine confidence estimation within these groups. This allows the model to dynamically adjust its calibration strategy without relying on dataset-specific prior knowledge about miscalibrated subgroups. Extensive experiments on real-world datasets demonstrate that our approach not only improves global calibration but also significantly enhances calibration within groups defined by feature similarity, topology, and connectivity, outperforming previous methods and demonstrating its effectiveness in practical scenarios.

Related papers

• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [53.79160907725975]
  We find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance.
  arXiv  Detail & Related papers  (2024-03-05T11:44:14Z)
• Towards Calibrated Deep Clustering Network [60.71776081164377]
  In deep clustering, the estimated confidence for a sample belonging to a particular cluster greatly exceeds its actual prediction accuracy. We propose a novel dual-head (calibration head and clustering head) deep clustering model that can effectively calibrate the estimated confidence and the actual accuracy. Extensive experiments demonstrate the proposed calibrated deep clustering model not only surpasses state-of-the-art deep clustering methods by 10 times in terms of expected calibration error but also significantly outperforms them in terms of clustering accuracy.
  arXiv  Detail & Related papers  (2024-03-04T11:23:40Z)
• Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration [16.930766717110053]
  We propose a novel meta-set-based cascaded temperature regression method for post-hoc calibration. We partition each meta-set into subgroups based on predicted category and confidence level, capturing diverse uncertainties. A regression network is then trained to derive category-specific and confidence-level-specific scaling, achieving calibration across meta-sets.
  arXiv  Detail & Related papers  (2024-02-14T14:35:57Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Beyond calibration: estimating the grouping loss of modern neural networks [68.8204255655161]
  Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
  arXiv  Detail & Related papers  (2022-10-28T07:04:20Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets [20.456742449675904]
  CCAC (Confidence with an Auxiliary Class) is a new post-hoc confidence calibration method for deep neural network (DNN) Key novelty of CCAC is an auxiliary class in the calibration model which separates mis-classified samples from correctly classified ones. Our experiments on different DNN models, datasets and applications show that CCAC can consistently outperform the prior post-hoc calibration methods.
  arXiv  Detail & Related papers  (2020-06-16T04:06:21Z)
• Calibrate and Prune: Improving Reliability of Lottery Tickets Through Prediction Calibration [40.203492372949576]
  Supervised models with uncalibrated confidences tend to be overconfident even when making wrong prediction. We study how explicit confidence calibration in the over- parameterized network impacts the quality of the resulting lottery tickets. Our empirical studies reveal that including calibration mechanisms consistently lead to more effective lottery tickets.
  arXiv  Detail & Related papers  (2020-02-10T15:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.