GDL-DS: A Benchmark for Geometric Deep Learning under Distribution Shifts

• URL: http://arxiv.org/abs/2310.08677v1
• Date: Thu, 12 Oct 2023 19:27:43 GMT
• Title: GDL-DS: A Benchmark for Geometric Deep Learning under Distribution Shifts
• Authors: Deyu Zou, Shikun Liu, Siqi Miao, Victor Fung, Shiyu Chang, Pan Li
• Abstract summary: GDL-DS is a benchmark designed for evaluating the performance of GDL models in scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics and materials science to biochemistry. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting.
• Score: 39.21363872039499
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Geometric deep learning (GDL) has gained significant attention in various scientific fields, chiefly for its proficiency in modeling data with intricate geometric structures. Yet, very few works have delved into its capability of tackling the distribution shift problem, a prevalent challenge in many relevant applications. To bridge this gap, we propose GDL-DS, a comprehensive benchmark designed for evaluating the performance of GDL models in scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics and materials science to biochemistry, and encapsulate a broad spectrum of distribution shifts including conditional, covariate, and concept shifts. Furthermore, we study three levels of information access from the out-of-distribution (OOD) testing data, including no OOD information, only OOD features without labels, and OOD features with a few labels. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting. A thorough analysis of the evaluation results is provided, poised to illuminate insights for DGL researchers and domain practitioners who are to use DGL in their applications.

Related papers

• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Survey [3.929140365559557]
  This paper provides a comprehensive overview of the latest techniques for understanding 3DPC using deep transfer learning (DTL) and domain adaptation (DA) The paper covers various applications, such as 3DPC object detection, semantic labeling, segmentation, classification, registration, downsampling/upsampling, and denoising.
  arXiv  Detail & Related papers  (2024-07-25T08:47:27Z)
• CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models [58.57987316300529]
  Large Language Models (LLMs) are increasingly deployed to handle various natural language processing (NLP) tasks. To evaluate the biases exhibited by LLMs, researchers have recently proposed a variety of datasets. We propose CEB, a Compositional Evaluation Benchmark that covers different types of bias across different social groups and tasks.
  arXiv  Detail & Related papers  (2024-07-02T16:31:37Z)
• Unifying Unsupervised Graph-Level Anomaly Detection and Out-of-Distribution Detection: A Benchmark [73.58840254552656]
  Unsupervised graph-level anomaly detection (GLAD) and unsupervised graph-level out-of-distribution (OOD) detection have received significant attention in recent years. We present a Unified Benchmark for unsupervised Graph-level OOD and anomaly Detection (our method) Our benchmark encompasses 35 datasets spanning four practical anomaly and OOD detection scenarios. We conduct multi-dimensional analyses to explore the effectiveness, generalizability, robustness, and efficiency of existing methods.
  arXiv  Detail & Related papers  (2024-06-21T04:07:43Z)
• On the performativity of SDG classifications in large bibliometric databases [0.0]
  Large bibliometric databases have taken up the UN's Sustainable Development Goals in their respective classifications. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data.
  arXiv  Detail & Related papers  (2024-05-05T17:28:54Z)
• Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks [60.82579717007963]
  We introduce an enhanced evaluation framework designed to more accurately gauge the effectiveness, consistency, and overall capability of Graph Contrastive Learning (GCL) methods.
  arXiv  Detail & Related papers  (2024-02-24T01:47:56Z)
• Do Deep Neural Networks Always Perform Better When Eating More Data? [82.6459747000664]
  We design experiments from Identically Independent Distribution(IID) and Out of Distribution(OOD) Under IID condition, the amount of information determines the effectivity of each sample, the contribution of samples and difference between classes determine the amount of class information. Under OOD condition, the cross-domain degree of samples determine the contributions, and the bias-fitting caused by irrelevant elements is a significant factor of cross-domain.
  arXiv  Detail & Related papers  (2022-05-30T15:40:33Z)
• Locally Adaptive Algorithms for Multiple Testing with Network Structure, with Application to Genome-Wide Association Studies [4.851566905442038]
  We propose a principled and generic framework for incorporating network data or multiple samples of auxiliary data from related source domains. LASLA employs a $p$-value weighting approach, utilizing structural insights to assign data-driven weights to individual test points. LASLA is illustrated through various synthetic experiments and an application to T2D-associated SNP identification.
  arXiv  Detail & Related papers  (2022-03-22T04:58:03Z)
• A Spectral-Spatial-Dependent Global Learning Framework for Insufficient and Imbalanced Hyperspectral Image Classification [16.93904035334754]
  spectral-spatial dependent global learning (SSDGL) framework based on global convolutional long short-term memory (GCL) and global joint attention mechanism (GJAM) SSDGL has powerful performance in insufficient and imbalanced sample problems and is superior to other state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-05-29T15:39:03Z)
• Applications of Unsupervised Deep Transfer Learning to Intelligent Fault Diagnosis: A Survey and Comparative Study [1.2345552555178128]
  We construct a new taxonomy and perform a comprehensive review of UDTL-based IFD according to different tasks. To emphasize the importance and importance of UDTL-based IFD, the whole test framework will be released to the research community.
  arXiv  Detail & Related papers  (2019-12-28T21:45:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.