Explaining Vision GNNs: A Semantic and Visual Analysis of Graph-based Image Classification

• URL: http://arxiv.org/abs/2504.19682v1
• Date: Mon, 28 Apr 2025 11:13:40 GMT
• Title: Explaining Vision GNNs: A Semantic and Visual Analysis of Graph-based Image Classification
• Authors: Nikolaos Chaidos, Angeliki Dimitriou, Nikolaos Spanos, Athanasios Voulodimos, Giorgos Stamou,
• Abstract summary: Graph Neural Networks (GNNs) have emerged as an efficient alternative to convolutional approaches for vision tasks.<n>Despite their efficiency, the explainability of GNN-based vision models remains underexplored.
• Score: 4.714421854862438
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Graph Neural Networks (GNNs) have emerged as an efficient alternative to convolutional approaches for vision tasks such as image classification, leveraging patch-based representations instead of raw pixels. These methods construct graphs where image patches serve as nodes, and edges are established based on patch similarity or classification relevance. Despite their efficiency, the explainability of GNN-based vision models remains underexplored, even though graphs are naturally interpretable. In this work, we analyze the semantic consistency of the graphs formed at different layers of GNN-based image classifiers, focusing on how well they preserve object structures and meaningful relationships. A comprehensive analysis is presented by quantifying the extent to which inter-layer graph connections reflect semantic similarity and spatial coherence. Explanations from standard and adversarial settings are also compared to assess whether they reflect the classifiers' robustness. Additionally, we visualize the flow of information across layers through heatmap-based visualization techniques, thereby highlighting the models' explainability. Our findings demonstrate that the decision-making processes of these models can be effectively explained, while also revealing that their reasoning does not necessarily align with human perception, especially in deeper layers.

Related papers

• i-WiViG: Interpretable Window Vision GNN [20.781355086144814]
  We tackle the self-interpretability of graph-based vision models by proposing our Interpretable Window Vision GNN (i-WiViG) approach.<n>This is achieved with window-based image graph processing that constrains the node receptive field to a local image region.<n>We evaluate our approach to remote sensing classification and regression tasks, showing it achieves competitive performance.
  arXiv  Detail & Related papers  (2025-03-11T11:31:40Z)
• Exploring Consistency in Graph Representations:from Graph Kernels to Graph Neural Networks [4.235378870514331]
  Graph Networks (GNNs) have emerged as a dominant approach in graph representation learning.<n>We bridge the gap between neural network methods and kernel approaches by enabling GNNs to consistently capture structures in their learned representations.<n>Inspired by these findings, we conjecture that the consistency in the similarities of graph representations across GNN layers is crucial in capturing relational structures and enhancing graph classification performance.
  arXiv  Detail & Related papers  (2024-10-31T09:07:08Z)
• Structure Your Data: Towards Semantic Graph Counterfactuals [1.8817715864806608]
  Counterfactual explanations (CEs) based on concepts are explanations that consider alternative scenarios to understand which high-level semantic features contributed to model predictions. In this work, we propose CEs based on the semantic graphs accompanying input data to achieve more descriptive, accurate, and human-aligned explanations.
  arXiv  Detail & Related papers  (2024-03-11T08:40:37Z)
• DGNN: Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations [62.04558318166396]
  Graph neural networks (GNNs) demonstrate a robust capability for representation learning on graphs with complex structures. A novel GNNs framework, dubbed Decoupled Graph Neural Networks (DGNN), is introduced to obtain a more comprehensive embedding representation of nodes. Experimental results conducted on several graph benchmark datasets verify DGNN's superiority in node classification task.
  arXiv  Detail & Related papers  (2024-01-28T06:43:13Z)
• On Discprecncies between Perturbation Evaluations of Graph Neural Network Attributions [49.8110352174327]
  We assess attribution methods from a perspective not previously explored in the graph domain: retraining. The core idea is to retrain the network on important (or not important) relationships as identified by the attributions. We run our analysis on four state-of-the-art GNN attribution methods and five synthetic and real-world graph classification datasets.
  arXiv  Detail & Related papers  (2024-01-01T02:03:35Z)
• Patch-wise Graph Contrastive Learning for Image Translation [69.85040887753729]
  We exploit the graph neural network to capture the topology-aware features. We construct the graph based on the patch-wise similarity from a pretrained encoder. In order to capture the hierarchical semantic structure, we propose the graph pooling.
  arXiv  Detail & Related papers  (2023-12-13T15:45:19Z)
• Learning Hierarchical Graph Representation for Image Manipulation Detection [50.04902159383709]
  The objective of image manipulation detection is to identify and locate the manipulated regions in the images. Recent approaches mostly adopt the sophisticated Convolutional Neural Networks (CNNs) to capture the tampering artifacts left in the images. We propose a hierarchical Graph Convolutional Network (HGCN-Net), which consists of two parallel branches.
  arXiv  Detail & Related papers  (2022-01-15T01:54:25Z)
• Inference Graphs for CNN Interpretation [12.765543440576144]
  Convolutional neural networks (CNNs) have achieved superior accuracy in many visual related tasks. We propose to model the network hidden layers activity using probabilistic models. We show that such graphs are useful for understanding the general inference process of a class, as well as explaining decisions the network makes regarding specific images.
  arXiv  Detail & Related papers  (2021-10-20T13:56:09Z)
• Graph Representation Learning for Spatial Image Steganalysis [11.358487655918678]
  We introduce a graph representation learning architecture for spatial image steganalysis. In the detailed architecture, we translate each image to a graph, where nodes represent the patches of the image and edges indicate the local associations between the patches. By feeding the graph to an attention network, the discriminative features can be learned for efficient steganalysis.
  arXiv  Detail & Related papers  (2021-10-03T09:09:08Z)
• Learning the Implicit Semantic Representation on Graph-Structured Data [57.670106959061634]
  Existing representation learning methods in graph convolutional networks are mainly designed by describing the neighborhood of each node as a perceptual whole. We propose a Semantic Graph Convolutional Networks (SGCN) that explores the implicit semantics by learning latent semantic-paths in graphs.
  arXiv  Detail & Related papers  (2021-01-16T16:18:43Z)
• Multi-Level Graph Convolutional Network with Automatic Graph Learning for Hyperspectral Image Classification [63.56018768401328]
  We propose a Multi-level Graph Convolutional Network (GCN) with Automatic Graph Learning method (MGCN-AGL) for HSI classification. By employing attention mechanism to characterize the importance among spatially neighboring regions, the most relevant information can be adaptively incorporated to make decisions. Our MGCN-AGL encodes the long range dependencies among image regions based on the expressive representations that have been produced at local level.
  arXiv  Detail & Related papers  (2020-09-19T09:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.