Understanding the Generalizability of Link Predictors Under Distribution Shifts on Graphs

• URL: http://arxiv.org/abs/2406.08788v1
• Date: Thu, 13 Jun 2024 03:47:12 GMT
• Title: Understanding the Generalizability of Link Predictors Under Distribution Shifts on Graphs
• Authors: Jay Revolinsky, Harry Shomer, Jiliang Tang,
• Abstract summary: Many popular benchmark datasets assume that dataset samples are drawn from the same distribution. We introduce LP-specific data splits which utilize structural properties to induce a controlled distribution shift. We verify the shift's effect empirically through evaluation of different SOTA LP methods and subsequently couple these methods with generalization techniques.
• Score: 34.58496513149175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, multiple models proposed for link prediction (LP) demonstrate impressive results on benchmark datasets. However, many popular benchmark datasets often assume that dataset samples are drawn from the same distribution (i.e., IID samples). In real-world situations, this assumption is often incorrect; since uncontrolled factors may lead train and test samples to come from separate distributions. To tackle the distribution shift problem, recent work focuses on creating datasets that feature distribution shifts and designing generalization methods that perform well on the new data. However, those studies only consider distribution shifts that affect {\it node-} and {\it graph-level} tasks, thus ignoring link-level tasks. Furthermore, relatively few LP generalization methods exist. To bridge this gap, we introduce a set of LP-specific data splits which utilizes structural properties to induce a controlled distribution shift. We verify the shift's effect empirically through evaluation of different SOTA LP methods and subsequently couple these methods with generalization techniques. Interestingly, LP-specific methods frequently generalize poorly relative to heuristics or basic GNN methods. Finally, this work provides analysis to uncover insights for enhancing LP generalization. Our code is available at: \href{https://github.com/revolins/LPStructGen}{https://github.com/revolins/LPStructGen}

Related papers

• FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection [24.969694113366216]
  Federated learning (FL) is a promising machine learning paradigm that collaborates with client models to capture global knowledge. deploying FL models in real-world scenarios remains unreliable due to the coexistence of in-distribution data and unexpected out-of-distribution data. We propose FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution.
  arXiv  Detail & Related papers  (2024-10-15T08:39:31Z)
• First-Order Manifold Data Augmentation for Regression Learning [4.910937238451485]
  We introduce FOMA: a new data-driven domain-independent data augmentation method. We evaluate FOMA on in-distribution generalization and out-of-distribution benchmarks, and we show that it improves the generalization of several neural architectures.
  arXiv  Detail & Related papers  (2024-06-16T12:35:05Z)
• Graph Out-of-Distribution Generalization via Causal Intervention [69.70137479660113]
  We introduce a conceptually simple yet principled approach for training robust graph neural networks (GNNs) under node-level distribution shifts. Our method resorts to a new learning objective derived from causal inference that coordinates an environment estimator and a mixture-of-expert GNN predictor. Our model can effectively enhance generalization with various types of distribution shifts and yield up to 27.4% accuracy improvement over state-of-the-arts on graph OOD generalization benchmarks.
  arXiv  Detail & Related papers  (2024-02-18T07:49:22Z)
• Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization [64.62570402941387]
  We use a single test sample to adapt multi-modal prompts at test time by minimizing the feature distribution shift to bridge the gap in the test domain. Our method improves zero-shot top- 1 accuracy beyond existing prompt-learning techniques, with a 3.08% improvement over the baseline MaPLe.
  arXiv  Detail & Related papers  (2023-11-02T17:59:32Z)
• Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time [69.77704012415845]
  Temporal shifts can considerably degrade performance of machine learning models deployed in the real world. We benchmark 13 prior approaches, including methods in domain generalization, continual learning, self-supervised learning, and ensemble learning. Under both evaluation strategies, we observe an average performance drop of 20% from in-distribution to out-of-distribution data.
  arXiv  Detail & Related papers  (2022-11-25T17:07:53Z)
• AnoShift: A Distribution Shift Benchmark for Unsupervised Anomaly Detection [7.829710051617368]
  We introduce an unsupervised anomaly detection benchmark with data that shifts over time, built over Kyoto-2006+, a traffic dataset for network intrusion detection. We first highlight the non-stationary nature of the data, using a basic per-feature analysis, t-SNE, and an Optimal Transport approach for measuring the overall distribution distances between years. We validate the performance degradation over time with diverse models, ranging from classical approaches to deep learning.
  arXiv  Detail & Related papers  (2022-06-30T17:59:22Z)
• Deep learning model solves change point detection for multiple change types [69.77452691994712]
  A change points detection aims to catch an abrupt disorder in data distribution. We propose an approach that works in the multiple-distributions scenario.
  arXiv  Detail & Related papers  (2022-04-15T09:44:21Z)
• Discovering Invariant Rationales for Graph Neural Networks [104.61908788639052]
  Intrinsic interpretability of graph neural networks (GNNs) is to find a small subset of the input graph's features. We propose a new strategy of discovering invariant rationale (DIR) to construct intrinsically interpretable GNNs.
  arXiv  Detail & Related papers  (2022-01-30T16:43:40Z)
• Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning [32.51394862932118]
  We present the ooDML benchmark to characterize generalization under out-of-distribution shifts in DML. ooDML is designed to probe the generalization performance on much more challenging, diverse train-to-test distribution shifts. We find that while generalization tends to consistently degrade with difficulty, some methods are better at retaining performance as the distribution shift increases.
  arXiv  Detail & Related papers  (2021-07-20T15:26:09Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• I-SPEC: An End-to-End Framework for Learning Transportable, Shift-Stable Models [6.802401545890963]
  Shifts in environment between development and deployment cause classical supervised learning to produce models that fail to generalize to new target distributions. We propose I-SPEC, an end-to-end framework that addresses this shortcoming by using data to learn a partial ancestral graph (PAG) We apply I-SPEC to a mortality prediction problem to show it can learn a model that is robust to shifts without needing upfront knowledge of the full causal DAG.
  arXiv  Detail & Related papers  (2020-02-20T18:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.