Transformer Interpretability Beyond Attention Visualization
- URL: http://arxiv.org/abs/2012.09838v2
- Date: Mon, 5 Apr 2021 11:19:28 GMT
- Title: Transformer Interpretability Beyond Attention Visualization
- Authors: Hila Chefer, Shir Gur, Lior Wolf
- Abstract summary: Self-attention techniques, and specifically Transformers, are dominating the field of text processing.
In this work, we propose a novel way to compute relevancy for Transformer networks.
- Score: 87.96102461221415
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Self-attention techniques, and specifically Transformers, are dominating the
field of text processing and are becoming increasingly popular in computer
vision classification tasks. In order to visualize the parts of the image that
led to a certain classification, existing methods either rely on the obtained
attention maps or employ heuristic propagation along the attention graph. In
this work, we propose a novel way to compute relevancy for Transformer
networks. The method assigns local relevance based on the Deep Taylor
Decomposition principle and then propagates these relevancy scores through the
layers. This propagation involves attention layers and skip connections, which
challenge existing methods. Our solution is based on a specific formulation
that is shown to maintain the total relevancy across layers. We benchmark our
method on very recent visual Transformer networks, as well as on a text
classification problem, and demonstrate a clear advantage over the existing
explainability methods.
Related papers
- DAPE V2: Process Attention Score as Feature Map for Length Extrapolation [63.87956583202729]
We conceptualize attention as a feature map and apply the convolution operator to mimic the processing methods in computer vision.
The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution.
arXiv Detail & Related papers (2024-10-07T07:21:49Z) - Analyzing Vision Transformers for Image Classification in Class
Embedding Space [5.210197476419621]
This work introduces a method to reverse-engineer Vision Transformers trained to solve image classification tasks.
Inspired by previous research in NLP, we demonstrate how the inner representations at any level of the hierarchy can be projected onto the learned class space.
We use our framework to show how image tokens develop class-specific representations that depend on attention mechanisms and contextual information.
arXiv Detail & Related papers (2023-10-29T10:25:23Z) - Towards Hierarchical Regional Transformer-based Multiple Instance
Learning [2.16656895298847]
We propose a Transformer-based multiple instance learning approach that replaces the traditional learned attention mechanism with a regional, Vision Transformer inspired self-attention mechanism.
We present a method that fuses regional patch information to derive slide-level predictions and show how this regional aggregation can be stacked to hierarchically process features on different distance levels.
Our approach is able to significantly improve performance over the baseline on two histopathology datasets and points towards promising directions for further research.
arXiv Detail & Related papers (2023-08-24T08:19:15Z) - Decoding Layer Saliency in Language Transformers [0.0]
In visual networks where saliency is more well-studied, saliency is naturally localized through the convolutional layers of the network.
We adapt gradient-based saliency methods for these networks, propose a method for evaluating the degree of semantic coherence of each layer, and demonstrate consistent improvement on multiple benchmark classification datasets.
arXiv Detail & Related papers (2023-08-09T20:53:22Z) - SATS: Self-Attention Transfer for Continual Semantic Segmentation [50.51525791240729]
continual semantic segmentation suffers from the same catastrophic forgetting issue as in continual classification learning.
This study proposes to transfer a new type of information relevant to knowledge, i.e. the relationships between elements within each image.
The relationship information can be effectively obtained from the self-attention maps in a Transformer-style segmentation model.
arXiv Detail & Related papers (2022-03-15T06:09:28Z) - Augmenting Convolutional networks with attention-based aggregation [55.97184767391253]
We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning.
We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth)
It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption.
arXiv Detail & Related papers (2021-12-27T14:05:41Z) - TransFG: A Transformer Architecture for Fine-grained Recognition [27.76159820385425]
Recently, vision transformer (ViT) shows its strong performance in the traditional classification task.
We propose a novel transformer-based framework TransFG where we integrate all raw attention weights of the transformer into an attention map.
A contrastive loss is applied to further enlarge the distance between feature representations of similar sub-classes.
arXiv Detail & Related papers (2021-03-14T17:03:53Z) - Visualization of Supervised and Self-Supervised Neural Networks via
Attribution Guided Factorization [87.96102461221415]
We develop an algorithm that provides per-class explainability.
In an extensive battery of experiments, we demonstrate the ability of our methods to class-specific visualization.
arXiv Detail & Related papers (2020-12-03T18:48:39Z) - CrossTransformers: spatially-aware few-shot transfer [92.33252608837947]
Given new tasks with very little data, modern vision systems degrade remarkably quickly.
We show how the neural network representations which underpin modern vision systems are subject to supervision collapse.
We propose self-supervised learning to encourage general-purpose features that transfer better.
arXiv Detail & Related papers (2020-07-22T15:37:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.