GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts

• URL: http://arxiv.org/abs/2310.08677v2
• Date: Tue, 19 Nov 2024 20:01:28 GMT
• Title: GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts
• Authors: Deyu Zou, Shikun Liu, Siqi Miao, Victor Fung, Shiyu Chang, Pan Li,
• Abstract summary: We propose GeSS, a benchmark designed for evaluating the performance of GDL models in scientific scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics, materials science to biochemistry, and encapsulate a broad spectrum of distribution shifts. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting.
• Score: 37.00741148951341
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Geometric deep learning (GDL) has gained significant attention in scientific fields, for its proficiency in modeling data with intricate geometric structures. However, very few works have delved into its capability of tackling the distribution shift problem, a prevalent challenge in many applications. To bridge this gap, we propose GeSS, a comprehensive benchmark designed for evaluating the performance of GDL models in scientific scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics, materials science to biochemistry, and encapsulate a broad spectrum of distribution shifts including conditional, covariate, and concept shifts. Furthermore, we study three levels of information access from the out-of-distribution (OOD) test data, including no OOD information, only unlabeled OOD data, and OOD data with a few labels. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting. A thorough analysis of the evaluation results is provided, poised to illuminate insights for GDL researchers and domain practitioners who are to use GDL in their applications.

Related papers

• Rebalancing the Scales: A Systematic Mapping Study of Generative Adversarial Networks (GANs) in Addressing Data Imbalance [0.16385815610837165]
  Generative Adrial Networks (GANs) showed immense potential as a data preprocessing technique that generates good quality synthetic data. This study employs a systematic mapping methodology to analyze 3041 papers on GAN-based sampling techniques for imbalanced data sourced from four digital libraries. Through comprehensive quantitative analysis, this research introduces three categorization mappings as application domains, GAN techniques, and GAN variants used to handle the imbalanced nature of the data.
  arXiv  Detail & Related papers  (2025-02-23T11:03:29Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? [58.330879414174476]
  We introduce DSBench, a benchmark designed to evaluate data science agents with realistic tasks. This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions. Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG)
  arXiv  Detail & Related papers  (2024-09-12T02:08:00Z)
• CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models [58.57987316300529]
  Large Language Models (LLMs) are increasingly deployed to handle various natural language processing (NLP) tasks. To evaluate the biases exhibited by LLMs, researchers have recently proposed a variety of datasets. We propose CEB, a Compositional Evaluation Benchmark that covers different types of bias across different social groups and tasks.
  arXiv  Detail & Related papers  (2024-07-02T16:31:37Z)
• On the performativity of SDG classifications in large bibliometric databases [0.0]
  Large bibliometric databases have taken up the UN's Sustainable Development Goals in their respective classifications. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data.
  arXiv  Detail & Related papers  (2024-05-05T17:28:54Z)
• SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research [11.816426823341134]
  We propose SciEval, a comprehensive and multi-disciplinary evaluation benchmark to address these issues. Based on Bloom's taxonomy, SciEval covers four dimensions to systematically evaluate scientific research ability. Both objective and subjective questions are included in SciEval.
  arXiv  Detail & Related papers  (2023-08-25T03:05:33Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• Do Deep Neural Networks Always Perform Better When Eating More Data? [82.6459747000664]
  We design experiments from Identically Independent Distribution(IID) and Out of Distribution(OOD) Under IID condition, the amount of information determines the effectivity of each sample, the contribution of samples and difference between classes determine the amount of class information. Under OOD condition, the cross-domain degree of samples determine the contributions, and the bias-fitting caused by irrelevant elements is a significant factor of cross-domain.
  arXiv  Detail & Related papers  (2022-05-30T15:40:33Z)
• Locally Adaptive Algorithms for Multiple Testing with Network Structure, with Application to Genome-Wide Association Studies [4.851566905442038]
  We propose a principled and generic framework for incorporating network data or multiple samples of auxiliary data from related source domains. LASLA employs a $p$-value weighting approach, utilizing structural insights to assign data-driven weights to individual test points. LASLA is illustrated through various synthetic experiments and an application to T2D-associated SNP identification.
  arXiv  Detail & Related papers  (2022-03-22T04:58:03Z)
• Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science [2.959890389883449]
  We introduce surrogate- and invariance-boosted contrastive learning (SIB-CL), a deep learning framework which incorporates three inexpensive'' and easily obtainable auxiliary information sources to overcome data scarcity. We demonstrate SIB-CL's effectiveness and generality on various scientific problems, e.g., predicting the density-of-states of 2D photonic crystals and solving the 3D time-independent Schrodinger equation.
  arXiv  Detail & Related papers  (2021-10-15T23:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.