Test-Time Training with KV Binding Is Secretly Linear Attention

• URL: http://arxiv.org/abs/2602.21204v2
• Date: Fri, 27 Feb 2026 15:30:32 GMT
• Title: Test-Time Training with KV Binding Is Secretly Linear Attention
• Abstract summary: Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning.<n>We show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator.
• Score: 30.99606181109631
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning that memorizes a key-value mapping at test time. However, our analysis reveals multiple phenomena that contradict this memorization-based interpretation. Motivated by these findings, we revisit the formulation of TTT and show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator. Beyond explaining previously puzzling model behaviors, this perspective yields multiple practical benefits: it enables principled architectural simplifications, admits fully parallel formulations that preserve performance while improving efficiency, and provides a systematic reduction of diverse TTT variants to a standard linear attention form. Overall, our results reframe TTT not as test-time memorization, but as learned linear attention with enhanced representational capacity.

Related papers

• Modular TTT: Rethinking Test-Time Training as Composable Modules [35.87120372962587]
  Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule.<n>Existing approaches typically hard-code each variant separately.<n>We propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph.
  arXiv  Detail & Related papers  (2026-08-07T11:11:43Z)
• Learning What to Remember: Test-Time Training via Context Distillation [72.10421333095852]
  Test-time training (TTT) is an appealing approach that performs online parameter updates for long-context modeling.<n>We propose textbfTest-textbfTime textbfContext textbfDistillation (TTCD), a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use.
  arXiv  Detail & Related papers  (2026-08-03T04:06:06Z)
• UniT: Unified Multimodal Chain-of-Thought Test-time Scaling [85.590774707406]
  Unified models can handle both multimodal understanding and generation within a single architecture, yet they typically operate in a single pass without iteratively refining their outputs.<n>We introduce UniT, a framework for multimodal test-time scaling that enables a single unified model to reason, verify, and refine across multiple rounds.
  arXiv  Detail & Related papers  (2026-02-12T18:59:49Z)
• ViT$^3$: Unlocking Test-Time Training in Vision [56.74014676094694]
  Test-Time Training (TTT) has emerged as a promising direction for efficient sequence modeling.<n>We present a systematic empirical study of TTT designs for visual sequence modeling.<n>We conclude with the Vision Test-Time Training (ViT$3$) model, a pure TTT architecture that achieves linear complexity and parallelizable computation.
  arXiv  Detail & Related papers  (2025-12-01T13:14:48Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models [64.02612380298228]
  Recent studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT)<n>We propose a model in which TTT achieves a substantially smaller in-distribution test error than global training.<n>We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet.
  arXiv  Detail & Related papers  (2025-09-29T09:24:52Z)
• CTA: Cross-Task Alignment for Better Test Time Training [10.54024648915477]
  Test-Time Training (TTT) has emerged as an effective method to enhance model robustness.<n>We introduce CTA (Cross-Task Alignment), a novel approach for improving TTT.<n>We show substantial improvements in robustness and generalization over the state-of-the-art on several benchmark datasets.
  arXiv  Detail & Related papers  (2025-07-07T17:33:20Z)
• Test-Time Training Provably Improves Transformers as In-context Learners [49.09821664572445]
  We investigate a gradient-based TTT algorithm for in-context learning.<n>We train a transformer model on the in-context demonstrations provided in the test prompt.<n>As our empirical contribution, we study the benefits of TTT for TabPFN.
  arXiv  Detail & Related papers  (2025-03-14T20:06:37Z)
• Efficient Test-Time Prompt Tuning for Vision-Language Models [41.90997623029582]
  Self-TPT is a framework leveraging Self-supervised learning for efficient Test-time Prompt Tuning. We show that Self-TPT not only significantly reduces inference costs but also achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-08-11T13:55:58Z)
• Test-Time Training on Graphs with Large Language Models (LLMs) [68.375487369596]
  Test-Time Training (TTT) has been proposed as a promising approach to train Graph Neural Networks (GNNs) Inspired by the great annotation ability of Large Language Models (LLMs) on Text-Attributed Graphs (TAGs), we propose to enhance the test-time training on graphs with LLMs as annotators. A two-stage training strategy is designed to tailor the test-time model with the limited and noisy labels.
  arXiv  Detail & Related papers  (2024-04-21T08:20:02Z)
• Residual Tensor Train: a Flexible and Efficient Approach for Learning Multiple Multilinear Correlations [4.754987078078158]
  In this paper, we present a novel Residual Train (ResTT) which integrates the merits of TT and residual structure. In particular, we prove that the fully-connected layer in neural networks and the Volterra series can be taken as special cases of ResTT. We prove that such a rule is much more relaxed than that of TT, which means ResTT can easily address the vanishing and exploding gradient problem.
  arXiv  Detail & Related papers  (2021-08-19T12:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.