Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs

• URL: http://arxiv.org/abs/2412.17609v1
• Date: Mon, 23 Dec 2024 14:28:56 GMT
• Title: Towards Foundation Models on Graphs: An Analysis on Cross-Dataset Transfer of Pretrained GNNs
• Authors: Fabrizio Frasca, Fabian Jogl, Moshe Eliasof, Matan Ostrovsky, Carola-Bibiane Schönlieb, Thomas Gärtner, Haggai Maron,
• Abstract summary: We study the extent to which pretrained Graph Neural Networks can be applied across datasets. We propose an extension to capture feature information while still being feature-agnostic.
• Score: 25.58407005007563
• License:
• Abstract: To develop a preliminary understanding towards Graph Foundation Models, we study the extent to which pretrained Graph Neural Networks can be applied across datasets, an effort requiring to be agnostic to dataset-specific features and their encodings. We build upon a purely structural pretraining approach and propose an extension to capture feature information while still being feature-agnostic. We evaluate pretrained models on downstream tasks for varying amounts of training samples and choices of pretraining datasets. Our preliminary results indicate that embeddings from pretrained models improve generalization only with enough downstream data points and in a degree which depends on the quantity and properties of pretraining data. Feature information can lead to improvements, but currently requires some similarities between pretraining and downstream feature spaces.

Related papers

• Strategies for Pretraining Neural Operators [5.812284760539713]
  Pretraining for partial differential equation (PDE) modeling has recently shown promise in scaling neural operators across datasets to improve generalizability and performance. We compare various pretraining methods without optimizing architecture choices to characterize pretraining dynamics. We find that pretraining is highly dependent on model and dataset choices, but in general transfer learning or physics-based pretraining strategies work best.
  arXiv  Detail & Related papers  (2024-06-12T17:56:46Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks [39.71761440499148]
  Pre-training on graph neural networks (GNNs) aims to learn transferable knowledge for downstream tasks with unlabeled data. We propose a better-with-less framework for graph pre-training: fewer, but carefully chosen data are fed into a GNN model. Experiment results show that the proposed APT is able to obtain an efficient pre-training model with fewer training data and better downstream performance.
  arXiv  Detail & Related papers  (2023-11-02T07:09:59Z)
• AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset [25.935496432142976]
  It is a long-term vision for Autonomous Driving (AD) community that the perception models can learn from a large-scale point cloud dataset. We formulate the point-cloud pre-training task as a semi-supervised problem, which leverages the few-shot labeled and massive unlabeled point-cloud data. We achieve significant performance gains on a series of downstream perception benchmarks including nuScenes, and KITTI, under different baseline models.
  arXiv  Detail & Related papers  (2023-06-01T12:32:52Z)
• When to Pre-Train Graph Neural Networks? From Data Generation Perspective! [19.239863500722983]
  Graph pre-training aims to acquire transferable knowledge from unlabeled graph data to improve downstream performance. This paper introduces a generic framework W2PGNN to answer the question of when to pre-train. W2PGNN offers three broad applications: providing the application scope of graph pre-trained models, the feasibility of pre-training, and assistance in selecting pre-training data to enhance downstream performance.
  arXiv  Detail & Related papers  (2023-03-29T05:05:02Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice [52.11183787786718]
  Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications. Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy. We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
  arXiv  Detail & Related papers  (2021-11-24T06:18:32Z)
• Unified Instance and Knowledge Alignment Pretraining for Aspect-based Sentiment Analysis [96.53859361560505]
  Aspect-based Sentiment Analysis (ABSA) aims to determine the sentiment polarity towards an aspect. There always exists severe domain shift between the pretraining and downstream ABSA datasets. We introduce a unified alignment pretraining framework into the vanilla pretrain-finetune pipeline.
  arXiv  Detail & Related papers  (2021-10-26T04:03:45Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.