Addressing Shortcomings in Fair Graph Learning Datasets: Towards a New Benchmark

• URL: http://arxiv.org/abs/2403.06017v2
• Date: Tue, 18 Jun 2024 03:55:04 GMT
• Title: Addressing Shortcomings in Fair Graph Learning Datasets: Towards a New Benchmark
• Authors: Xiaowei Qian, Zhimeng Guo, Jialiang Li, Haitao Mao, Bingheng Li, Suhang Wang, Yao Ma,
• Abstract summary: We develop and introduce a collection of synthetic, semi-synthetic, and real-world datasets that fulfill a broad spectrum of requirements. These datasets are thoughtfully designed to include relevant graph structures and bias information crucial for the fair evaluation of models. Our extensive experimental results with fair graph learning methods across our datasets demonstrate their effectiveness in benchmarking the performance of these methods.
• Score: 26.233696733521757
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fair graph learning plays a pivotal role in numerous practical applications. Recently, many fair graph learning methods have been proposed; however, their evaluation often relies on poorly constructed semi-synthetic datasets or substandard real-world datasets. In such cases, even a basic Multilayer Perceptron (MLP) can outperform Graph Neural Networks (GNNs) in both utility and fairness. In this work, we illustrate that many datasets fail to provide meaningful information in the edges, which may challenge the necessity of using graph structures in these problems. To address these issues, we develop and introduce a collection of synthetic, semi-synthetic, and real-world datasets that fulfill a broad spectrum of requirements. These datasets are thoughtfully designed to include relevant graph structures and bias information crucial for the fair evaluation of models. The proposed synthetic and semi-synthetic datasets offer the flexibility to create data with controllable bias parameters, thereby enabling the generation of desired datasets with user-defined bias values with ease. Moreover, we conduct systematic evaluations of these proposed datasets and establish a unified evaluation approach for fair graph learning models. Our extensive experimental results with fair graph learning methods across our datasets demonstrate their effectiveness in benchmarking the performance of these methods. Our datasets and the code for reproducing our experiments are available at https://github.com/XweiQ/Benchmark-GraphFairness.

Related papers

• Graph Reasoning Networks [9.18586425686959]
  Graph Reasoning Networks (GRNs) is a novel approach to combine the strengths of fixed and learned graph representations and a reasoning module based on a differentiable satisfiability solver. Results on real-world datasets show comparable performance to GNNs. Experiments on synthetic datasets demonstrate the potential of the newly proposed method.
  arXiv  Detail & Related papers  (2024-07-08T10:53:49Z)
• Rethinking the Effectiveness of Graph Classification Datasets in Benchmarks for Assessing GNNs [7.407592553310068]
  We propose an empirical protocol based on a fair benchmarking framework to investigate the performance discrepancy between simple methods and GNNs. We also propose a novel metric to quantify the dataset effectiveness by considering both dataset complexity and model performance. Our findings shed light on the current understanding of benchmark datasets, and our new platform could fuel the future evolution of graph classification benchmarks.
  arXiv  Detail & Related papers  (2024-07-06T08:33:23Z)
• Deceptive Fairness Attacks on Graphs via Meta Learning [102.53029537886314]
  We study deceptive fairness attacks on graphs to answer the question: How can we achieve poisoning attacks on a graph learning model to exacerbate the bias deceptively? We propose a meta learning-based framework named FATE to attack various fairness definitions and graph learning models. We conduct extensive experimental evaluations on real-world datasets in the task of semi-supervised node classification.
  arXiv  Detail & Related papers  (2023-10-24T09:10:14Z)
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
  We present a code that successfully replicates results from six popular and recent graph recommendation models. We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
  arXiv  Detail & Related papers  (2023-08-01T09:31:44Z)
• GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networks [72.01829954658889]
  This paper introduces the mathematical definition of this novel problem setting. We devise a general framework that coordinates a single graph-shared structure learner and multiple graph-specific GNNs. The well-trained structure learner can directly produce adaptive structures for unseen target graphs without any fine-tuning.
  arXiv  Detail & Related papers  (2023-06-20T03:33:22Z)
• Diving into Unified Data-Model Sparsity for Class-Imbalanced Graph Representation Learning [30.23894624193583]
  Graph Neural Networks (GNNs) training upon non-Euclidean graph data often encounters relatively higher time costs. We develop a unified data-model dynamic sparsity framework named Graph Decantation (GraphDec) to address challenges brought by training upon a massive class-imbalanced graph data.
  arXiv  Detail & Related papers  (2022-10-01T01:47:00Z)
• Analyzing the Effect of Sampling in GNNs on Individual Fairness [79.28449844690566]
  Graph neural network (GNN) based methods have saturated the field of recommender systems. We extend an existing method for promoting individual fairness on graphs to support mini-batch, or sub-sample based, training of a GNN. We show that mini-batch training facilitate individual fairness promotion by allowing for local nuance to guide the process of fairness promotion in representation learning.
  arXiv  Detail & Related papers  (2022-09-08T16:20:25Z)
• Evaluating Explainability for Graph Neural Networks [21.339111121529815]
  We introduce a synthetic graph data generator, ShapeGGen, which can generate a variety of benchmark datasets. We include ShapeGGen and several real-world graph datasets into an open-source graph explainability library, GraphXAI.
  arXiv  Detail & Related papers  (2022-08-19T13:43:52Z)
• A Graph-Enhanced Click Model for Web Search [67.27218481132185]
  We propose a novel graph-enhanced click model (GraphCM) for web search. We exploit both intra-session and inter-session information for the sparsity and cold-start problems.
  arXiv  Detail & Related papers  (2022-06-17T08:32:43Z)
• Bias Reduction via Cooperative Bargaining in Synthetic Graph Dataset Generation [1.6942548626426182]
  We propose a method to find a synthetic graph dataset that has an even representation of graphs with different metrics. The resulting dataset can then be used, among others, for benchmarking graph processing techniques.
  arXiv  Detail & Related papers  (2022-05-27T11:12:50Z)
• Optimal Propagation for Graph Neural Networks [51.08426265813481]
  We propose a bi-level optimization approach for learning the optimal graph structure. We also explore a low-rank approximation model for further reducing the time complexity.
  arXiv  Detail & Related papers  (2022-05-06T03:37:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.