Do Attention Heads Compete or Cooperate during Counting?

• URL: http://arxiv.org/abs/2502.06923v1
• Date: Mon, 10 Feb 2025 17:21:39 GMT
• Title: Do Attention Heads Compete or Cooperate during Counting?
• Authors: Pál Zsámboki, Ádám Fraknói, Máté Gedeon, András Kornai, Zsolt Zombori,
• Abstract summary: We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting.<n>We ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction.
• Score: 0.12116854758481393
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present an in-depth mechanistic interpretability analysis of training small transformers on an elementary task, counting, which is a crucial deductive step in many algorithms. In particular, we investigate the collaboration/competition among the attention heads: we ask whether the attention heads behave as a pseudo-ensemble, all solving the same subtask, or they perform different subtasks, meaning that they can only solve the original task in conjunction. Our work presents evidence that on the semantics of the counting task, attention heads behave as a pseudo-ensemble, but their outputs need to be aggregated in a non-uniform manner in order to create an encoding that conforms to the syntax. Our source code will be available upon publication.

Related papers

• Head Pursuit: Probing Attention Specialization in Multimodal Transformers [32.218423952797444]
  We study how individual attention heads in text-generative models specialize in specific semantic or visual attributes.<n>Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers.<n>Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output.
  arXiv  Detail & Related papers  (2025-10-24T14:41:47Z)
• $K$-MSHC: Unmasking Minimally Sufficient Head Circuits in Large Language Models with Experiments on Syntactic Classification Tasks [3.767957313558699]
  We introduce the $(bmK, epsilon)$-Minimum Sufficient Head Circuit, a methodology to identify minimal sets of attention heads crucial for classification tasks.<n>Applying our Search-K-MSHC algorithm to Gemma-9B, we analyze three syntactic task families: grammar acceptability, arithmetic verification, and arithmetic word problems.<n>Our findings reveal distinct task-specific head circuits, with grammar tasks predominantly utilizing early layers, word problems showing pronounced activity in both shallow and deep regions, and arithmetic verification demonstrating a more distributed pattern across the network.
  arXiv  Detail & Related papers  (2025-05-18T07:15:01Z)
• Attend or Perish: Benchmarking Attention in Algorithmic Reasoning [0.0]
  We propose AttentionSpan, an algorithmic benchmark comprising five tasks of infinite input domains.<n>This allows us to assess (i) models' ability to extrapolate to unseen types of inputs, including new lengths, value ranges or input domains, but also (ii)to assess the robustness of their learned mechanisms.
  arXiv  Detail & Related papers  (2025-02-28T22:50:38Z)
• What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks [15.874604623294427]
  We show a transformer with only one attention layer can excel in memorization but falls short in other tasks. We identify a class of simple operations that a single attention layer can execute, and show that the complex tasks can be approached as the combinations of these simple operations.
  arXiv  Detail & Related papers  (2024-04-02T02:45:12Z)
• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [56.222097640468306]
  We provide mechanistic understanding of how transformers learn "semantic structure" We show, through a combination of mathematical analysis and experiments on Wikipedia data, that the embedding layer and the self-attention layer encode the topical structure.
  arXiv  Detail & Related papers  (2023-03-07T21:42:17Z)
• Continual Learning with Distributed Optimization: Does CoCoA Forget? [0.0]
  We focus on the continual learning problem where the tasks arrive sequentially. The aim is to perform well on the newly arrived task without performance degradation on the previously seen tasks. We consider the well-established distributed learning algorithm COCOA.
  arXiv  Detail & Related papers  (2022-11-30T13:49:43Z)
• Coarse-to-Fine: Hierarchical Multi-task Learning for Natural Language Understanding [51.31622274823167]
  We propose a hierarchical framework with a coarse-to-fine paradigm, with the bottom level shared to all the tasks, the mid-level divided to different groups, and the top-level assigned to each of the tasks. This allows our model to learn basic language properties from all tasks, boost performance on relevant tasks, and reduce the negative impact from irrelevant tasks.
  arXiv  Detail & Related papers  (2022-08-19T02:46:20Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Pretext Tasks selection for multitask self-supervised speech representation learning [23.39079406674442]
  This paper introduces a method to select a group of pretext tasks among a set of candidates. Experiments conducted on speaker recognition and automatic speech recognition validate our approach.
  arXiv  Detail & Related papers  (2021-07-01T16:36:29Z)
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study [75.42182503265056]
  Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm. We deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks.
  arXiv  Detail & Related papers  (2021-05-08T22:26:52Z)
• What's in your Head? Emergent Behaviour in Multi-Task Transformer Models [26.557793822750302]
  We study the behaviour of non-target heads, that is, the output of heads when given input that belongs to a different task than the one they were trained for. We find that non-target heads exhibit emergent behaviour, which may either explain the target task, or generalize beyond their original task.
  arXiv  Detail & Related papers  (2021-04-13T12:04:30Z)
• The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT [18.13834903235249]
  Multi-headed attention heads are a mainstay in transformer-based models. Different methods have been proposed to classify the role of each attention head based on the relations between tokens which have high pair-wise attention. We formalize a simple yet effective score that generalizes to all the roles of attention heads and employs hypothesis testing on this score for robust inference.
  arXiv  Detail & Related papers  (2021-01-22T14:10:59Z)
• Learning Hard Retrieval Decoder Attention for Transformers [69.40942736249397]
  Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding.
  arXiv  Detail & Related papers  (2020-09-30T13:18:57Z)
• Robust Learning Through Cross-Task Consistency [92.42534246652062]
  We propose a broadly applicable and fully computational method for augmenting learning with Cross-Task Consistency. We observe that learning with cross-task consistency leads to more accurate predictions and better generalization to out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-06-07T09:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.