Human Attention in Fine-grained Classification

• URL: http://arxiv.org/abs/2111.01628v1
• Date: Tue, 2 Nov 2021 14:41:11 GMT
• Title: Human Attention in Fine-grained Classification
• Authors: Yao Rong, Wenjia Xu, Zeynep Akata, Enkelejda Kasneci
• Abstract summary: We validate that human attention contains valuable information for decision-making processes such as fine-grained classification. We propose Gaze Augmentation Training (GAT) and Knowledge Fusion Network (KFN) to integrate human gaze into classification models.
• Score: 38.71613202835921
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The way humans attend to, process and classify a given image has the potential to vastly benefit the performance of deep learning models. Exploiting where humans are focusing can rectify models when they are deviating from essential features for correct decisions. To validate that human attention contains valuable information for decision-making processes such as fine-grained classification, we compare human attention and model explanations in discovering important features. Towards this goal, we collect human gaze data for the fine-grained classification dataset CUB and build a dataset named CUB-GHA (Gaze-based Human Attention). Furthermore, we propose the Gaze Augmentation Training (GAT) and Knowledge Fusion Network (KFN) to integrate human gaze knowledge into classification models. We implement our proposals in CUB-GHA and the recently released medical dataset CXR-Eye of chest X-ray images, which includes gaze data collected from a radiologist. Our result reveals that integrating human attention knowledge benefits classification effectively, e.g. improving the baseline by 4.38% on CXR. Hence, our work provides not only valuable insights into understanding human attention in fine-grained classification, but also contributes to future research in integrating human gaze with computer vision tasks. CUB-GHA and code are available at https://github.com/yaorong0921/CUB-GHA.

Related papers

Err

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.