Modular TTT: Rethinking Test-Time Training as Composable Modules

• URL: http://arxiv.org/abs/2608.07110v1
• Date: Fri, 07 Aug 2026 11:11:43 GMT
• Title: Modular TTT: Rethinking Test-Time Training as Composable Modules
• Abstract summary: Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule.<n>Existing approaches typically hard-code each variant separately.<n>We propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph.
• Score: 35.87120372962587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.

Related papers

• In-Place Test-Time Training [32.521599123691026]
  In-Place Test-Time Training (In-Place TTT) is a framework that seamlessly endows Large Language Models with Test-Time Training.<n>In-Place TTT treats the final projection matrix of the ubiquitous blocks as its adaptable fast weights.<n>We replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective aligned with the Next-Token-Prediction task.
  arXiv  Detail & Related papers  (2026-04-07T17:59:44Z)
• Test-Time Training with KV Binding Is Secretly Linear Attention [30.99606181109631]
  Test-time training (TTT) with KV binding as sequence modeling layer is commonly interpreted as a form of online meta-learning.<n>We show that a broad class of TTT architectures can be expressed as a form of learned linear attention operator.
  arXiv  Detail & Related papers  (2026-02-24T18:59:30Z)
• TNT: Improving Chunkwise Training for Test-Time Memorization [62.78875147721906]
  Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers.<n>We introduce TNT, a novel training paradigm that decouples training efficiency from inference performance through a two-stage process.<n>TNT achieves a substantial acceleration in training speed-up to 17 times faster than the most accurate baseline configuration.
  arXiv  Detail & Related papers  (2025-11-10T17:45:09Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• Test-Time Training Provably Improves Transformers as In-context Learners [49.09821664572445]
  We investigate a gradient-based TTT algorithm for in-context learning.<n>We train a transformer model on the in-context demonstrations provided in the test prompt.<n>As our empirical contribution, we study the benefits of TTT for TabPFN.
  arXiv  Detail & Related papers  (2025-03-14T20:06:37Z)
• Enhancing Plasticity for First Session Adaptation Continual Learning [20.62749699589017]
  We introduce Plasticity-Enhanced Test-Time Adaptation in Class-Incremental Learning (PLASTIC)<n>PLASTIC reinstates plasticity in CIL while preserving model stability.<n>It consistently outperforms both conventional and state-of-the-art PTM-based CIL approaches.
  arXiv  Detail & Related papers  (2023-10-17T13:06:39Z)
• Test-Time Training on Video Streams [66.63237260332984]
  Prior work has established Test-Time Training (TTT) as a general framework to further improve a trained model at test time.<n>We extend TTT to the streaming setting, where multiple test instances arrive in temporal order.<n>Online TTT significantly outperforms the fixed-model baseline for four tasks, on three real-world datasets.
  arXiv  Detail & Related papers  (2023-07-11T05:17:42Z)
• Decouple Graph Neural Networks: Train Multiple Simple GNNs Simultaneously Instead of One [60.5818387068983]
  Graph neural networks (GNN) suffer from severe inefficiency. We propose to decouple a multi-layer GNN as multiple simple modules for more efficient training. We show that the proposed framework is highly efficient with reasonable performance.
  arXiv  Detail & Related papers  (2023-04-20T07:21:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.