Temporal Dependencies in In-Context Learning: The Role of Induction Heads

• URL: http://arxiv.org/abs/2604.01094v1
• Date: Wed, 01 Apr 2026 16:21:38 GMT
• Title: Temporal Dependencies in In-Context Learning: The Role of Induction Heads
• Abstract summary: We show that large language models consistently display a serial-recall-like pattern, assigning peak probability to tokens that immediately follow a repeated token in the input sequence.<n>We show that induction heads, specialized attention heads that attend to the token following a previous occurrence of the current token, play an important role in this phenomenon.
• Score: 4.433315630787157
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) exhibit strong in-context learning capabilities, but how they track and retrieve information from context remains underexplored. Drawing on the free recall paradigm in cognitive science (where participants recall list items in any order), we show that several open-source LLMs consistently display a serial-recall-like pattern, assigning peak probability to tokens that immediately follow a repeated token in the input sequence. Through systematic ablation experiments, we show that induction heads, specialized attention heads that attend to the token following a previous occurrence of the current token, play an important role in this phenomenon. Removing heads with a high induction score substantially reduces the +1 lag bias, whereas ablating random heads does not reproduce the same reduction. We also show that removing heads with high induction scores impairs the performance of models prompted to do serial recall using few-shot learning to a larger extent than removing random heads. Our findings highlight a mechanistically specific connection between induction heads and temporal context processing in transformers, suggesting that these heads are especially important for ordered retrieval and serial-recall-like behavior during in-context learning.

Related papers

• You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differences [71.56121289529226]
  We introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video.<n>TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation.<n>Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks.
  arXiv  Detail & Related papers  (2026-06-14T18:34:54Z)
• Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models [24.666925550391024]
  We identify induction heads as a key driver of the repetition curse.<n>We propose a technique with attention head regularization that could be employed to reduce the dominance of induction heads during generation.
  arXiv  Detail & Related papers  (2025-05-17T03:09:33Z)
• Demystifying Verbatim Memorization in Large Language Models [67.49068128909349]
  Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications. We develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences. We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to memorize verbatim sequences, even for out-of-distribution sequences.
  arXiv  Detail & Related papers  (2024-07-25T07:10:31Z)
• From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty [67.81977289444677]
  Large language models (LLMs) often exhibit undesirable behaviors, such as hallucinations and sequence repetitions.<n>We categorize fallback behaviors - sequence repetitions, degenerate text, and hallucinations - and extensively analyze them.<n>Our experiments reveal a clear and consistent ordering of fallback behaviors, across all these axes.
  arXiv  Detail & Related papers  (2024-07-08T16:13:42Z)
• Retrieval Head Mechanistically Explains Long-Context Factuality [56.78951509492645]
  We show that a special type of attention heads are largely responsible for retrieving information, which we dub retrieval heads. We show that retrieval heads strongly influence chain-of-thought (CoT) reasoning, where the model needs to frequently refer back the question and previously-generated context. We believe our insights will foster future research on reducing hallucination, improving reasoning, and compressing the KV cache.
  arXiv  Detail & Related papers  (2024-04-24T00:24:03Z)
• Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective [91.14291142262262]
  This work presents a straightforward and fundamental explanation from the data perspective. Our preliminary investigation reveals a strong correlation between the degeneration issue and the presence of repetitions in training data. Our experiments reveal that penalizing the repetitions in training data remains critical even when considering larger model sizes and instruction tuning.
  arXiv  Detail & Related papers  (2023-10-16T09:35:42Z)
• In-context Learning and Induction Heads [5.123049926855312]
  "Induction heads" are attention heads that implement a simple algorithm to complete token sequences. We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability.
  arXiv  Detail & Related papers  (2022-09-24T00:43:19Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.