End-to-End Test-Time Training for Long Context

• URL: http://arxiv.org/abs/2512.23675v2
• Date: Wed, 31 Dec 2025 18:41:09 GMT
• Title: End-to-End Test-Time Training for Long Context
• Abstract summary: We formulate long-context language modeling as a problem in continual learning rather than architecture design.<n>Our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights.<n>Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning)
• Score: 98.3930777591529
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.

Related papers

• Rethinking Expressivity and Efficiency in Test-Time Training [26.484618002837635]
  Test-Time Training (TTT) enables long-context processing via continuous weight updates during inference.<n>We propose E$2$-TTT to balance the expressivity of per-token update dynamics with the hardware efficiency of chunk-wise approximations.<n>E$2$-TTT can match the training throughput of efficient chunk-wise methods, demonstrating that it effectively reconciles expressivity with efficiency.
  arXiv  Detail & Related papers  (2026-08-21T17:12:28Z)
• Self-Guided Test-Time Training for Long-Context LLMs [16.160114043456215]
  Long-context processing is increasingly important for large language models.<n>As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question.<n>A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation.<n>We propose Self-Guided TTT (S-TTT), which identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans.
  arXiv  Detail & Related papers  (2026-07-10T13:45:56Z)
• TTA-Vid: Generalized Test-Time Adaptation for Video Reasoning [54.70019148172847]
  We leverage the paradigm of Test-Time Reinforcement Learning on video-language data to allow for adapting a pretrained model to incoming video samples at test-time without explicit labels.<n>The proposed test-time adaptation for video approach (TTA-Vid) combines two components that work simultaneously.<n>TTA-Vid yields consistent improvements across various video reasoning tasks and is able to outperform current state-of-the-art methods trained on large-scale data.
  arXiv  Detail & Related papers  (2026-04-01T09:52:57Z)
• Understanding the Role of Training Data in Test-Time Scaling [56.12341509545198]
  We study the performance of test-time scaling for transformers trained on an in-context weight prediction task for linear regression.<n>We show that training on a diverse, relevant, and hard set of tasks results in best performance for test-time scaling.
  arXiv  Detail & Related papers  (2025-10-04T01:38:48Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time [4.737679362712655]
  ourmodelacronym(Extend at Test-Time) is a method for extending the context length of short context Transformer-based Language Models.<n>We evaluate ETT on LongBench by extending the context length of GPT-Large and Phi-2 up to 32 times, increasing from 1k to 32k tokens.
  arXiv  Detail & Related papers  (2025-07-08T18:06:45Z)
• Test-Time Training Done Right [61.8429380523577]
  Test-Time Training (TTT) models context by adapting part of the model's weights (referred to as fast weights) during inference.<n>Existing TTT methods struggled to show effectiveness in handling long-context data.<n>We develop Large Chunk Test-Time Training (LaCT) which improves hardware utilization by orders of magnitude.
  arXiv  Detail & Related papers  (2025-05-29T17:50:34Z)
• llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length [1.5857828218932415]
  We present llm-jp-modernbert, a ModernBERT model trained on a publicly available, massive Japanese corpus with a context length of 8192 tokens.<n>While our model does not surpass existing baselines on downstream tasks, it achieves good results on fill-mask test evaluations.
  arXiv  Detail & Related papers  (2025-04-22T02:45:19Z)
• How to Train Long-Context Language Models (Effectively) [75.5418485597276]
  We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information.<n>We find that code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short-context data.<n>Our final model, ProLong-8B, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K.
  arXiv  Detail & Related papers  (2024-10-03T16:46:52Z)
• Learning to (Learn at Test Time): RNNs with Expressive Hidden States [82.35503195691463]
  We present a framework for instantiating sequence modeling layers with linear complexity and expressive hidden states.<n>Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training layers.<n>Similar to Transformer, TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context.
  arXiv  Detail & Related papers  (2024-07-05T16:23:20Z)
• Test-Time Training on Video Streams [66.63237260332984]
  Prior work has established Test-Time Training (TTT) as a general framework to further improve a trained model at test time.<n>We extend TTT to the streaming setting, where multiple test instances arrive in temporal order.<n>Online TTT significantly outperforms the fixed-model baseline for four tasks, on three real-world datasets.
  arXiv  Detail & Related papers  (2023-07-11T05:17:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.