GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification

• URL: http://arxiv.org/abs/2305.18221v3
• Date: Tue, 29 Aug 2023 20:52:57 GMT
• Title: GazeGNN: A Gaze-Guided Graph Neural Network for Chest X-ray Classification
• Authors: Bin Wang, Hongyi Pan, Armstrong Aboah, Zheyuan Zhang, Elif Keles, Drew Torigian, Baris Turkbey, Elizabeth Krupinski, Jayaram Udupa, Ulas Bagci
• Abstract summary: We propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into visual attention maps (VAMs) We develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature.
• Score: 9.266556662553345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Eye tracking research is important in computer vision because it can help us understand how humans interact with the visual world. Specifically for high-risk applications, such as in medical imaging, eye tracking can help us to comprehend how radiologists and other medical professionals search, analyze, and interpret images for diagnostic and clinical purposes. Hence, the application of eye tracking techniques in disease classification has become increasingly popular in recent years. Contemporary works usually transform gaze information collected by eye tracking devices into visual attention maps (VAMs) to supervise the learning process. However, this is a time-consuming preprocessing step, which stops us from applying eye tracking to radiologists' daily work. To solve this problem, we propose a novel gaze-guided graph neural network (GNN), GazeGNN, to leverage raw eye-gaze data without being converted into VAMs. In GazeGNN, to directly integrate eye gaze into image classification, we create a unified representation graph that models both images and gaze pattern information. With this benefit, we develop a real-time, real-world, end-to-end disease classification algorithm for the first time in the literature. This achievement demonstrates the practicality and feasibility of integrating real-time eye tracking techniques into the daily work of radiologists. To our best knowledge, GazeGNN is the first work that adopts GNN to integrate image and eye-gaze data. Our experiments on the public chest X-ray dataset show that our proposed method exhibits the best classification performance compared to existing methods. The code is available at https://github.com/ukaukaaaa/GazeGNN.

Related papers

• GEM: Context-Aware Gaze EstiMation with Visual Search Behavior Matching for Chest Radiograph [32.1234295417225]
  We propose a context-aware Gaze EstiMation (GEM) network that utilizes eye gaze data collected from radiologists to simulate their visual search behavior patterns. It consists of a context-awareness module, visual behavior graph construction, and visual behavior matching. Experiments on four publicly available datasets demonstrate the superiority of GEM over existing methods.
  arXiv  Detail & Related papers  (2024-08-10T09:46:25Z)
• Multimodal Learning and Cognitive Processes in Radiology: MedGaze for Chest X-ray Scanpath Prediction [10.388541520456714]
  Our proposed system aims to predict eye gaze sequences from radiology reports and CXR images. Our model predicts fixation coordinates and durations critical for medical scanpath prediction, outperforming existing models in the computer vision community. Based on the radiologist's evaluation, MedGaze can generate human-like gaze sequences with a high focus on relevant regions.
  arXiv  Detail & Related papers  (2024-06-28T06:38:58Z)
• Gaze-directed Vision GNN for Mitigating Shortcut Learning in Medical Image [6.31072075551707]
  We propose a novel gaze-directed Vision GNN (called GD-ViG) to leverage the visual patterns of radiologists from gaze as expert knowledge. The experiments on two public medical image datasets demonstrate that GD-ViG outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-06-20T07:16:41Z)
• Graph Neural Networks for Brain Graph Learning: A Survey [53.74244221027981]
  Graph neural networks (GNNs) have demonstrated a significant advantage in mining graph-structured data. GNNs to learn brain graph representations for brain disorder analysis has recently gained increasing attention. In this paper, we aim to bridge this gap by reviewing brain graph learning works that utilize GNNs.
  arXiv  Detail & Related papers  (2024-06-01T02:47:39Z)
• A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective [71.03621840455754]
  Graph Neural Networks (GNNs) have gained momentum in graph representation learning. graph Transformers embed a graph structure into the Transformer architecture to overcome the limitations of local neighborhood aggregation. This paper presents a comprehensive review of GNNs and graph Transformers in computer vision from a task-oriented perspective.
  arXiv  Detail & Related papers  (2022-09-27T08:10:14Z)
• A Deep Learning Approach for the Segmentation of Electroencephalography Data in Eye Tracking Applications [56.458448869572294]
  We introduce DETRtime, a novel framework for time-series segmentation of EEG data. Our end-to-end deep learning-based framework brings advances in Computer Vision to the forefront. Our model generalizes well in the task of EEG sleep stage segmentation.
  arXiv  Detail & Related papers  (2022-06-17T10:17:24Z)
• Rectify ViT Shortcut Learning by Visual Saliency [40.55418820114868]
  Shortcut learning is common but harmful to deep learning models. In this work, we propose a novel and effective saliency-guided vision transformer (SGT) model to rectify shortcut learning.
  arXiv  Detail & Related papers  (2022-06-17T05:54:07Z)
• Evidential Temporal-aware Graph-based Social Event Detection via Dempster-Shafer Theory [76.4580340399321]
  We propose ETGNN, a novel Evidential Temporal-aware Graph Neural Network. We construct view-specific graphs whose nodes are the texts and edges are determined by several types of shared elements respectively. Considering the view-specific uncertainty, the representations of all views are converted into mass functions through evidential deep learning (EDL) neural networks.
  arXiv  Detail & Related papers  (2022-05-24T16:22:40Z)
• Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis [54.60796004113496]
  We demonstrate that the eye movement of radiologists reading medical images can be a new form of supervision to train the DNN-based computer-aided diagnosis (CAD) system. We record the tracks of the radiologists' gaze when they are reading images. The gaze information is processed and then used to supervise the DNN's attention via an Attention Consistency module.
  arXiv  Detail & Related papers  (2022-04-06T08:31:05Z)
• Node Masking: Making Graph Neural Networks Generalize and Scale Better [71.51292866945471]
  Graph Neural Networks (GNNs) have received a lot of interest in the recent times. In this paper, we utilize some theoretical tools to better visualize the operations performed by state of the art spatial GNNs. We introduce a simple concept, Node Masking, that allows them to generalize and scale better.
  arXiv  Detail & Related papers  (2020-01-17T06:26:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.