On Calibration of Graph Neural Networks for Node Classification

• URL: http://arxiv.org/abs/2206.01570v1
• Date: Fri, 3 Jun 2022 13:48:10 GMT
• Title: On Calibration of Graph Neural Networks for Node Classification
• Authors: Tong Liu, Yushan Liu, Marcel Hildebrandt, Mitchell Joblin, Hang Li, Volker Tresp
• Abstract summary: Graph neural networks learn entity and edge embeddings for tasks such as node classification and link prediction. These models achieve good performance with respect to accuracy, but the confidence scores associated with the predictions might not be calibrated. We propose a topology-aware calibration method that takes the neighboring nodes into account and yields improved calibration.
• Score: 29.738179864433445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graphs can model real-world, complex systems by representing entities and their interactions in terms of nodes and edges. To better exploit the graph structure, graph neural networks have been developed, which learn entity and edge embeddings for tasks such as node classification and link prediction. These models achieve good performance with respect to accuracy, but the confidence scores associated with the predictions might not be calibrated. That means that the scores might not reflect the ground-truth probabilities of the predicted events, which would be especially important for safety-critical applications. Even though graph neural networks are used for a wide range of tasks, the calibration thereof has not been sufficiently explored yet. We investigate the calibration of graph neural networks for node classification, study the effect of existing post-processing calibration methods, and analyze the influence of model capacity, graph density, and a new loss function on calibration. Further, we propose a topology-aware calibration method that takes the neighboring nodes into account and yields improved calibration compared to baseline methods.

Related papers

• Gradient Rewiring for Editable Graph Neural Network Training [84.77778876113099]
  underlineGradient underlineRewiring method for underlineEditable graph neural network training, named textbfGRE. We propose a simple yet effective underlineGradient underlineRewiring method for underlineEditable graph neural network training, named textbfGRE.
  arXiv  Detail & Related papers  (2024-10-21T01:01:50Z)
• GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networks [8.505932176266368]
  Graph Neural Networks deliver strong classification results but often suffer from poor calibration performance, leading to overconfidence or underconfidence. Existing post hoc methods, such as temperature scaling, fail to effectively utilize graph structures, while current GNN calibration methods often overlook the potential of leveraging diverse input information and model ensembles jointly. In the paper, we propose Graph Ensemble TemperatureScaling, a novel calibration framework that combines input and model ensemble strategies within a Graph Mixture of Experts archi SOTA calibration techniques, reducing expected calibration error by 25 percent across 10 GNN benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-12T15:34:41Z)
• Graph Mining under Data scarcity [6.229055041065048]
  We propose an Uncertainty Estimator framework that can be applied on top of any generic Graph Neural Networks (GNNs) We train these models under the classic episodic learning paradigm in the $n$-way, $k$-shot fashion, in an end-to-end setting. Our method outperforms the baselines, which demonstrates the efficacy of the Uncertainty Estimator for Few-shot node classification on graphs with a GNN.
  arXiv  Detail & Related papers  (2024-06-07T10:50:03Z)
• Decoupling Feature Extraction and Classification Layers for Calibrated Neural Networks [3.5284544394841117]
  We show that decoupling the training of feature extraction layers and classification layers in over-parametrized DNN architectures significantly improves model calibration. We illustrate these methods improve calibration across ViT and WRN architectures for several image classification benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-02T11:36:17Z)
• Online GNN Evaluation Under Test-time Graph Distribution Shifts [92.4376834462224]
  A new research problem, online GNN evaluation, aims to provide valuable insights into the well-trained GNNs's ability to generalize to real-world unlabeled graphs. We develop an effective learning behavior discrepancy score, dubbed LeBeD, to estimate the test-time generalization errors of well-trained GNN models.
  arXiv  Detail & Related papers  (2024-03-15T01:28:08Z)
• NCTV: Neural Clamping Toolkit and Visualization for Neural Network Calibration [66.22668336495175]
  A lack of consideration for neural network calibration will not gain trust from humans. We introduce the Neural Clamping Toolkit, the first open-source framework designed to help developers employ state-of-the-art model-agnostic calibrated models.
  arXiv  Detail & Related papers  (2022-11-29T15:03:05Z)
• What Makes Graph Neural Networks Miscalibrated? [48.00374886504513]
  We conduct a systematic study on the calibration qualities of graph neural networks (GNNs) We identify five factors which influence the calibration of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity. We design a novel calibration method named Graph Attention Temperature Scaling (GATS), which is tailored for calibrating graph neural networks.
  arXiv  Detail & Related papers  (2022-10-12T16:41:42Z)
• Bayesian Neural Network Versus Ex-Post Calibration For Prediction Uncertainty [0.2343856409260935]
  Probabilistic predictions from neural networks account for predictive uncertainty during classification. In practice most datasets are trained on non-probabilistic neural networks which by default do not capture this inherent uncertainty. A plausible alternative to the calibration approach is to use Bayesian neural networks, which directly models a predictive distribution.
  arXiv  Detail & Related papers  (2022-09-29T07:22:19Z)
• Distributionally Robust Semi-Supervised Learning Over Graphs [68.29280230284712]
  Semi-supervised learning (SSL) over graph-structured data emerges in many network science applications. To efficiently manage learning over graphs, variants of graph neural networks (GNNs) have been developed recently. Despite their success in practice, most of existing methods are unable to handle graphs with uncertain nodal attributes. Challenges also arise due to distributional uncertainties associated with data acquired by noisy measurements. A distributionally robust learning framework is developed, where the objective is to train models that exhibit quantifiable robustness against perturbations.
  arXiv  Detail & Related papers  (2021-10-20T14:23:54Z)
• Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks [54.23874144090228]
  A common approach is to learn a post-hoc calibration function that transforms the output of the original network into calibrated confidence scores. Previous post-hoc calibration techniques work only with simple calibration functions. We propose a new neural network architecture that represents a class of intra order-preserving functions.
  arXiv  Detail & Related papers  (2020-03-15T12:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.