VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

• URL: http://arxiv.org/abs/2305.13417v2
• Date: Fri, 24 Nov 2023 12:02:13 GMT
• Title: VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers
• Authors: Shahar Katz, Yonatan Belinkov
• Abstract summary: Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models to their vocabulary. We investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. We create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph.
• Score: 45.42482446288144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in interpretability suggest we can project weights and hidden states of transformer-based language models (LMs) to their vocabulary, a transformation that makes them more human interpretable. In this paper, we investigate LM attention heads and memory values, the vectors the models dynamically create and recall while processing a given input. By analyzing the tokens they represent through this projection, we identify patterns in the information flow inside the attention mechanism. Based on our discoveries, we create a tool to visualize a forward pass of Generative Pre-trained Transformers (GPTs) as an interactive flow graph, with nodes representing neurons or hidden states and edges representing the interactions between them. Our visualization simplifies huge amounts of data into easy-to-read plots that can reflect the models' internal processing, uncovering the contribution of each component to the models' final prediction. Our visualization also unveils new insights about the role of layer norms as semantic filters that influence the models' output, and about neurons that are always activated during forward passes and act as regularization vectors.

Related papers

• Talking Heads: Understanding Inter-layer Communication in Transformer Language Models [32.2976613483151]
  We analyze a mechanism used in two LMs to selectively inhibit items in a context in one task. We find that models write into low-rank subspaces of the residual stream to represent features which are then read out by later layers.
  arXiv  Detail & Related papers  (2024-06-13T18:12:01Z)
• Vision Transformers Need Registers [26.63912173005165]
  We identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. We show that this solution fixes that problem entirely for both supervised and self-supervised models.
  arXiv  Detail & Related papers  (2023-09-28T16:45:46Z)
• On the Transition from Neural Representation to Symbolic Knowledge [2.2528422603742304]
  We propose a Neural-Symbolic Transitional Dictionary Learning (TDL) framework that employs an EM algorithm to learn a transitional representation of data. We implement the framework with a diffusion model by regarding the decomposition of input as a cooperative game. We additionally use RL enabled by the Markovian of diffusion models to further tune the learned prototypes.
  arXiv  Detail & Related papers  (2023-08-03T19:29:35Z)
• AttentionViz: A Global View of Transformer Attention [60.82904477362676]
  We present a new visualization technique designed to help researchers understand the self-attention mechanism in transformers. The main idea behind our method is to visualize a joint embedding of the query and key vectors used by transformer models to compute attention. We create an interactive visualization tool, AttentionViz, based on these joint query-key embeddings.
  arXiv  Detail & Related papers  (2023-05-04T23:46:49Z)
• Holistically Explainable Vision Transformers [136.27303006772294]
  We propose B-cos transformers, which inherently provide holistic explanations for their decisions. Specifically, we formulate each model component - such as the multi-layer perceptrons, attention layers, and the tokenisation module - to be dynamic linear. We apply our proposed design to Vision Transformers (ViTs) and show that the resulting models, dubbed Bcos-ViTs, are highly interpretable and perform competitively to baseline ViTs.
  arXiv  Detail & Related papers  (2023-01-20T16:45:34Z)
• Wide and Narrow: Video Prediction from Context and Motion [54.21624227408727]
  We propose a new framework to integrate these complementary attributes to predict complex pixel dynamics through deep networks. We present global context propagation networks that aggregate the non-local neighboring representations to preserve the contextual information over the past frames. We also devise local filter memory networks that generate adaptive filter kernels by storing the motion of moving objects in the memory.
  arXiv  Detail & Related papers  (2021-10-22T04:35:58Z)
• Incorporating Residual and Normalization Layers into Analysis of Masked Language Models [29.828669678974983]
  We extend the scope of the analysis of Transformers from solely the attention patterns to the whole attention block. Our analysis of Transformer-based masked language models shows that the token-to-token interaction performed via attention has less impact on the intermediate representations than previously assumed.
  arXiv  Detail & Related papers  (2021-09-15T08:32:20Z)
• TCL: Transformer-based Dynamic Graph Modelling via Contrastive Learning [87.38675639186405]
  We propose a novel graph neural network approach, called TCL, which deals with the dynamically-evolving graph in a continuous-time fashion. To the best of our knowledge, this is the first attempt to apply contrastive learning to representation learning on dynamic graphs.
  arXiv  Detail & Related papers  (2021-05-17T15:33:25Z)
• VisBERT: Hidden-State Visualizations for Transformers [66.86452388524886]
  We present VisBERT, a tool for visualizing the contextual token representations within BERT for the task of (multi-hop) Question Answering. VisBERT enables users to get insights about the model's internal state and to explore its inference steps or potential shortcomings.
  arXiv  Detail & Related papers  (2020-11-09T15:37:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.