Human Interpretation and Exploitation of Self-attention Patterns in Transformers: A Case Study in Extractive Summarization

• URL: http://arxiv.org/abs/2112.05364v1
• Date: Fri, 10 Dec 2021 07:15:09 GMT
• Title: Human Interpretation and Exploitation of Self-attention Patterns in Transformers: A Case Study in Extractive Summarization
• Authors: Raymond Li, Wen Xiao, Lanjun Wang, Giuseppe Carenini
• Abstract summary: This paper synergize two lines of research in a human-in-the-loop pipeline to first find important task-specific attention patterns. Then those patterns are applied, not only to the original model, but also to smaller models. Experiments indicate that when we inject such patterns, both the original and the smaller model show improvements in performance and arguably interpretability.
• Score: 9.42402875164615
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The transformer multi-head self-attention mechanism has been thoroughly investigated recently. On one hand, researchers are interested in understanding why and how transformers work. On the other hand, they propose new attention augmentation methods to make transformers more accurate, efficient and interpretable. In this paper, we synergize these two lines of research in a human-in-the-loop pipeline to first find important task-specific attention patterns. Then those patterns are applied, not only to the original model, but also to smaller models, as a human-guided knowledge distillation process. The benefits of our pipeline are demonstrated in a case study with the extractive summarization task. After finding three meaningful attention patterns in the popular BERTSum model, experiments indicate that when we inject such patterns, both the original and the smaller model show improvements in performance and arguably interpretability.

Related papers

• Incremental Learning of Sparse Attention Patterns in Transformers [29.54151079577767]
  This paper introduces a high-order Markov chain task to investigate how transformers learn to integrate information from multiple past positions.<n>We identify a shift in learning dynamics from competitive, where heads converge on the most statistically dominant pattern, to cooperative, where heads specialize in distinct patterns.
  arXiv  Detail & Related papers  (2026-02-22T12:16:06Z)
• Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models [12.112842686827669]
  Transformer based models have shown remarkable capabilities in sequence learning across a wide range of tasks.<n>We investigate the layerwise behavior of Transformers to uncover the mechanisms underlying their multi-task generalization ability.<n>Our explicit constructions align closely with empirical observations, providing theoretical support for the Transformer's effectiveness and efficiency on sequence learning across diverse tasks.
  arXiv  Detail & Related papers  (2025-06-02T17:39:31Z)
• Linking forward-pass dynamics in Transformers and real-time human processing [6.165163123577484]
  We investigate the link between real-time processing in humans and "layer-time" dynamics in Transformer models. Our results suggest that Transformer processing and human processing may be facilitated or impeded by similar properties of an input stimulus.
  arXiv  Detail & Related papers  (2025-04-18T23:38:14Z)
• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [63.66763657191476]
  We show that efficient numerical training and inference algorithms as low-rank computation have impressive performance for learning Transformer-based adaption. We analyze how magnitude-based models affect generalization while improving adaption. We conclude that proper magnitude-based has a slight on the testing performance.
  arXiv  Detail & Related papers  (2024-06-24T23:00:58Z)
• All Roads Lead to Rome? Exploring the Invariance of Transformers' Representations [69.3461199976959]
  We propose a model based on invertible neural networks, BERT-INN, to learn the Bijection Hypothesis. We show the advantage of BERT-INN both theoretically and through extensive experiments.
  arXiv  Detail & Related papers  (2023-05-23T22:30:43Z)
• Evaluating Prompt-based Question Answering for Object Prediction in the Open Research Knowledge Graph [0.0]
  This work reports results on adopting prompt-based training of transformers for textitscholarly knowledge graph object prediction It deviates from the other works proposing entity and relation extraction pipelines for predicting objects of a scholarly knowledge graph. We find that (i) per expectations, transformer models when tested out-of-the-box underperform on a new domain of data, (ii) prompt-based training of the models achieve performance boosts of up to 40% in a relaxed evaluation setting.
  arXiv  Detail & Related papers  (2023-05-22T10:35:18Z)
• AttentionViz: A Global View of Transformer Attention [60.82904477362676]
  We present a new visualization technique designed to help researchers understand the self-attention mechanism in transformers. The main idea behind our method is to visualize a joint embedding of the query and key vectors used by transformer models to compute attention. We create an interactive visualization tool, AttentionViz, based on these joint query-key embeddings.
  arXiv  Detail & Related papers  (2023-05-04T23:46:49Z)
• How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers [59.57128476584361]
  We introduce PAPA, a new probing method that replaces the input-dependent attention matrices with constant ones. We find that without any input-dependent attention, all models achieve competitive performance. We show that better-performing models lose more from applying our method than weaker models, suggesting that the utilization of the input-dependent attention mechanism might be a factor in their success.
  arXiv  Detail & Related papers  (2022-11-07T12:37:54Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer [41.44769642537572]
  Unary-Pairwise Transformer is a two-stage detector that exploits unary and pairwise representations for HOIs. We evaluate our method on the HICO-DET and V-COCO datasets, and significantly outperform state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-12-03T10:52:06Z)
• On the Bias Against Inductive Biases [34.10348216388905]
  Self-supervised feature learning for visual tasks has seen state-of-the-art success using these extremely deep, isotropic networks. In this work, we analyze the effect of inductive biases on small to moderately-sized isotropic networks used for unsupervised visual feature learning.
  arXiv  Detail & Related papers  (2021-05-28T19:41:48Z)
• SparseBERT: Rethinking the Importance Analysis in Self-attention [107.68072039537311]
  Transformer-based models are popular for natural language processing (NLP) tasks due to its powerful capacity. Attention map visualization of a pre-trained model is one direct method for understanding self-attention mechanism. We propose a Differentiable Attention Mask (DAM) algorithm, which can be also applied in guidance of SparseBERT design.
  arXiv  Detail & Related papers  (2021-02-25T14:13:44Z)
• Guiding Attention for Self-Supervised Learning with Transformers [24.785500242464646]
  We propose a technique to allow for efficient self-supervised learning with bi-directional Transformers. Our approach is motivated by recent studies demonstrating that self-attention patterns in trained models contain a majority of non-linguistic regularities.
  arXiv  Detail & Related papers  (2020-10-06T00:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.