Online GNN Evaluation Under Test-time Graph Distribution Shifts

• URL: http://arxiv.org/abs/2403.09953v1
• Date: Fri, 15 Mar 2024 01:28:08 GMT
• Title: Online GNN Evaluation Under Test-time Graph Distribution Shifts
• Authors: Xin Zheng, Dongjin Song, Qingsong Wen, Bo Du, Shirui Pan,
• Abstract summary: A new research problem, online GNN evaluation, aims to provide valuable insights into the well-trained GNNs's ability to generalize to real-world unlabeled graphs. We develop an effective learning behavior discrepancy score, dubbed LeBeD, to estimate the test-time generalization errors of well-trained GNN models.
• Score: 92.4376834462224
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating the performance of a well-trained GNN model on real-world graphs is a pivotal step for reliable GNN online deployment and serving. Due to a lack of test node labels and unknown potential training-test graph data distribution shifts, conventional model evaluation encounters limitations in calculating performance metrics (e.g., test error) and measuring graph data-level discrepancies, particularly when the training graph used for developing GNNs remains unobserved during test time. In this paper, we study a new research problem, online GNN evaluation, which aims to provide valuable insights into the well-trained GNNs's ability to effectively generalize to real-world unlabeled graphs under the test-time graph distribution shifts. Concretely, we develop an effective learning behavior discrepancy score, dubbed LeBeD, to estimate the test-time generalization errors of well-trained GNN models. Through a novel GNN re-training strategy with a parameter-free optimality criterion, the proposed LeBeD comprehensively integrates learning behavior discrepancies from both node prediction and structure reconstruction perspectives. This enables the effective evaluation of the well-trained GNNs' ability to capture test node semantics and structural representations, making it an expressive metric for estimating the generalization error in online GNN evaluation. Extensive experiments on real-world test graphs under diverse graph distribution shifts could verify the effectiveness of the proposed method, revealing its strong correlation with ground-truth test errors on various well-trained GNN models.

Related papers

• GOODAT: Towards Test-time Graph Out-of-Distribution Detection [103.40396427724667]
  Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains. Recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN. This paper introduces a data-centric, unsupervised, and plug-and-play solution that operates independently of training data and modifications of GNN architecture.
  arXiv  Detail & Related papers  (2024-01-10T08:37:39Z)
• Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networks [40.95782849532316]
  We propose a novel training framework designed to improve intrinsic GNN uncertainty estimates. Our framework adapts the principle of centering data to graph data through novel graph anchoring strategies. Our work provides insights into uncertainty estimation for GNNs, and demonstrates the utility of G-$Delta$UQ in obtaining reliable estimates.
  arXiv  Detail & Related papers  (2024-01-07T00:58:33Z)
• Learning to Reweight for Graph Neural Network [63.978102332612906]
  Graph Neural Networks (GNNs) show promising results for graph tasks. Existing GNNs' generalization ability will degrade when there exist distribution shifts between testing and training graph data. We propose a novel nonlinear graph decorrelation method, which can substantially improve the out-of-distribution generalization ability.
  arXiv  Detail & Related papers  (2023-12-19T12:25:10Z)
• GNNEvaluator: Evaluating GNN Performance On Unseen Graphs Without Labels [81.93520935479984]
  We study a new problem, GNN model evaluation, that aims to assess the performance of a specific GNN model trained on labeled and observed graphs. We propose a two-stage GNN model evaluation framework, including (1) DiscGraph set construction and (2) GNNEvaluator training and inference. Under the effective training supervision from the DiscGraph set, GNNEvaluator learns to precisely estimate node classification accuracy of the to-be-evaluated GNN model.
  arXiv  Detail & Related papers  (2023-10-23T05:51:59Z)
• FRGNN: Mitigating the Impact of Distribution Shift on Graph Neural Networks via Test-Time Feature Reconstruction [13.21683198528012]
  A distribution shift can adversely affect the test performance of Graph Neural Networks (GNNs) We propose FR-GNN, a general framework for GNNs to conduct feature reconstruction. Notably, the reconstructed node features can be directly utilized for testing the well-trained model.
  arXiv  Detail & Related papers  (2023-08-18T02:34:37Z)
• Addressing the Impact of Localized Training Data in Graph Neural Networks [0.0]
  Graph Neural Networks (GNNs) have achieved notable success in learning from graph-structured data. This article aims to assess the impact of training GNNs on localized subsets of the graph. We propose a regularization method to minimize distributional discrepancies between localized training data and graph inference.
  arXiv  Detail & Related papers  (2023-07-24T11:04:22Z)
• Stable Prediction on Graphs with Agnostic Distribution Shift [105.12836224149633]
  Graph neural networks (GNNs) have been shown to be effective on various graph tasks with randomly separated training and testing data. In real applications, however, the distribution of training graph might be different from that of the test one. We propose a novel stable prediction framework for GNNs, which permits both locally and globally stable learning and prediction on graphs.
  arXiv  Detail & Related papers  (2021-10-08T02:45:47Z)
• Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training data [52.771780951404565]
  Shift-Robust GNN (SR-GNN) is designed to account for distributional differences between biased training data and the graph's true inference distribution. We show that SR-GNN outperforms other GNN baselines by accuracy, eliminating at least (40%) of the negative effects introduced by biased training data.
  arXiv  Detail & Related papers  (2021-08-02T18:00:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.