No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets

• URL: http://arxiv.org/abs/2502.02379v1
• Date: Tue, 04 Feb 2025 14:59:03 GMT
• Title: No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets
• Authors: Corinna Coupette, Jeremy Wayland, Emily Simons, Bastian Rieck,
• Abstract summary: We introduce RINGS, a flexible and mode-perturbation framework to assess the quality of graph-learning datasets. Within this framework, we propose two measures -- performance separability and mode complementarity -- as evaluation tools. We derive actionable recommendations for improving the evaluation of graph-learning methods.
• Score: 16.040478092904632
• License:
• Abstract: Benchmark datasets have proved pivotal to the success of graph learning, and good benchmark datasets are crucial to guide the development of the field. Recent research has highlighted problems with graph-learning datasets and benchmarking practices -- revealing, for example, that methods which ignore the graph structure can outperform graph-based approaches on popular benchmark datasets. Such findings raise two questions: (1) What makes a good graph-learning dataset, and (2) how can we evaluate dataset quality in graph learning? Our work addresses these questions. As the classic evaluation setup uses datasets to evaluate models, it does not apply to dataset evaluation. Hence, we start from first principles. Observing that graph-learning datasets uniquely combine two modes -- the graph structure and the node features -- , we introduce RINGS, a flexible and extensible mode-perturbation framework to assess the quality of graph-learning datasets based on dataset ablations -- i.e., by quantifying differences between the original dataset and its perturbed representations. Within this framework, we propose two measures -- performance separability and mode complementarity -- as evaluation tools, each assessing, from a distinct angle, the capacity of a graph dataset to benchmark the power and efficacy of graph-learning methods. We demonstrate the utility of our framework for graph-learning dataset evaluation in an extensive set of experiments and derive actionable recommendations for improving the evaluation of graph-learning methods. Our work opens new research directions in data-centric graph learning, and it constitutes a first step toward the systematic evaluation of evaluations.

Related papers

• Rethinking the Effectiveness of Graph Classification Datasets in Benchmarks for Assessing GNNs [7.407592553310068]
  We propose an empirical protocol based on a fair benchmarking framework to investigate the performance discrepancy between simple methods and GNNs. We also propose a novel metric to quantify the dataset effectiveness by considering both dataset complexity and model performance. Our findings shed light on the current understanding of benchmark datasets, and our new platform could fuel the future evolution of graph classification benchmarks.
  arXiv  Detail & Related papers  (2024-07-06T08:33:23Z)
• ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization [32.19963543411396]
  This study constructs a large-scale dataset of comprehensive chart-caption pairs and fine-tuning instructions on each chart. We propose an innovative chart summarization method, ChartThinker, which synthesizes deep analysis based on chains of thought. Built upon the curated datasets, our trained model consistently exhibits superior performance in chart summarization tasks.
  arXiv  Detail & Related papers  (2024-03-17T14:49:09Z)
• Addressing Shortcomings in Fair Graph Learning Datasets: Towards a New Benchmark [26.233696733521757]
  We develop and introduce a collection of synthetic, semi-synthetic, and real-world datasets that fulfill a broad spectrum of requirements. These datasets are thoughtfully designed to include relevant graph structures and bias information crucial for the fair evaluation of models. Our extensive experimental results with fair graph learning methods across our datasets demonstrate their effectiveness in benchmarking the performance of these methods.
  arXiv  Detail & Related papers  (2024-03-09T21:33:26Z)
• Towards Data-centric Graph Machine Learning: Review and Outlook [120.64417630324378]
  We introduce a systematic framework, Data-centric Graph Machine Learning (DC-GML), that encompasses all stages of the graph data lifecycle. A thorough taxonomy of each stage is presented to answer three critical graph-centric questions. We pinpoint the future prospects of the DC-GML domain, providing insights to navigate its advancements and applications.
  arXiv  Detail & Related papers  (2023-09-20T00:40:13Z)
• Bures-Wasserstein Means of Graphs [60.42414991820453]
  We propose a novel framework for defining a graph mean via embeddings in the space of smooth graph signal distributions. By finding a mean in this embedding space, we can recover a mean graph that preserves structural information. We establish the existence and uniqueness of the novel graph mean, and provide an iterative algorithm for computing it.
  arXiv  Detail & Related papers  (2023-05-31T11:04:53Z)
• InfoGCL: Information-Aware Graph Contrastive Learning [26.683911257080304]
  We study how graph information is transformed and transferred during the contrastive learning process. We propose an information-aware graph contrastive learning framework called InfoGCL. We show for the first time that all recent graph contrastive learning methods can be unified by our framework.
  arXiv  Detail & Related papers  (2021-10-28T21:10:39Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Effective and Efficient Graph Learning for Multi-view Clustering [173.8313827799077]
  We propose an effective and efficient graph learning model for multi-view clustering. Our method exploits the view-similar between graphs of different views by the minimization of tensor Schatten p-norm. Our proposed algorithm is time-economical and obtains the stable results and scales well with the data size.
  arXiv  Detail & Related papers  (2021-08-15T13:14:28Z)
• Group Contrastive Self-Supervised Learning on Graphs [101.45974132613293]
  We study self-supervised learning on graphs using contrastive methods. We argue that contrasting graphs in multiple subspaces enables graph encoders to capture more abundant characteristics.
  arXiv  Detail & Related papers  (2021-07-20T22:09:21Z)
• Model-Agnostic Graph Regularization for Few-Shot Learning [60.64531995451357]
  We present a comprehensive study on graph embedded few-shot learning. We introduce a graph regularization approach that allows a deeper understanding of the impact of incorporating graph information between labels. Our approach improves the performance of strong base learners by up to 2% on Mini-ImageNet and 6.7% on ImageNet-FS.
  arXiv  Detail & Related papers  (2021-02-14T05:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.