Learning What to Remember: Test-Time Training via Context Distillation

• URL: http://arxiv.org/abs/2608.01672v1
• Date: Mon, 03 Aug 2026 04:06:06 GMT
• Title: Learning What to Remember: Test-Time Training via Context Distillation
• Abstract summary: Test-time training (TTT) is an appealing approach that performs online parameter updates for long-context modeling.<n>We propose textbfTest-textbfTime textbfContext textbfDistillation (TTCD), a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use.
• Score: 72.10421333095852
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Effective long-context modeling is not merely about retaining more of the past, but about preserving the information that may prove relevant later. Test-time training (TTT) is an appealing approach that performs online parameter updates for long-context modeling, yet existing TTT methods only optimize either reconstruction or online adaptation objectives without considering the future utility of retained information. In this work, we propose \textbf{T}est-\textbf{T}ime \textbf{C}ontext \textbf{D}istillation (TTCD), a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use. Specifically, TTCD uses a long-window teacher to supervise the fast weights of a short-window student, where the hidden-state discrepancy between them offers a dense, self-supervised signal guiding the model to memorize the contextual information crucial for future token predictions. We focus on an in-place variant: In-Place TTCD (IP-TTCD), which uses the existing MLP parameters as the fast weights. Experiments on long-context language modeling tasks show IP-TTCD consistently outperforms DeltaNet, Gated DeltaNet, sliding-window attention, and TTT when pre-trained from scratch. Furthermore, IP-TTCD allows pre-trained transformer models to adapt their parameters during inference through continual pre-training, gaining long-context capabilities with only a lightweight architectural augmentation. Our results position TTCD as a step toward architectural continual learning.

Related papers

• Self-Guided Test-Time Training for Long-Context LLMs [16.160114043456215]
  Long-context processing is increasingly important for large language models.<n>As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question.<n>A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation.<n>We propose Self-Guided TTT (S-TTT), which identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans.
  arXiv  Detail & Related papers  (2026-07-10T13:45:56Z)
• In-Place Test-Time Training [32.521599123691026]
  In-Place Test-Time Training (In-Place TTT) is a framework that seamlessly endows Large Language Models with Test-Time Training.<n>In-Place TTT treats the final projection matrix of the ubiquitous blocks as its adaptable fast weights.<n>We replace TTT's generic reconstruction objective with a tailored, theoretically-grounded objective aligned with the Next-Token-Prediction task.
  arXiv  Detail & Related papers  (2026-04-07T17:59:44Z)
• Test time training enhances in-context learning of nonlinear functions [51.56484100374058]
  Test-time training (TTT) enhances model performance by explicitly updating designated parameters prior to each prediction.<n>We investigate the combination of TTT with in-context learning (ICL), where the model is given a few examples from the target distribution at inference time.
  arXiv  Detail & Related papers  (2025-09-30T03:56:44Z)
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models [64.02612380298228]
  Recent studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT)<n>We propose a model in which TTT achieves a substantially smaller in-distribution test error than global training.<n>We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet.
  arXiv  Detail & Related papers  (2025-09-29T09:24:52Z)
• CTA: Cross-Task Alignment for Better Test Time Training [10.54024648915477]
  Test-Time Training (TTT) has emerged as an effective method to enhance model robustness.<n>We introduce CTA (Cross-Task Alignment), a novel approach for improving TTT.<n>We show substantial improvements in robustness and generalization over the state-of-the-art on several benchmark datasets.
  arXiv  Detail & Related papers  (2025-07-07T17:33:20Z)
• Test-Time Training Provably Improves Transformers as In-context Learners [49.09821664572445]
  We investigate a gradient-based TTT algorithm for in-context learning.<n>We train a transformer model on the in-context demonstrations provided in the test prompt.<n>As our empirical contribution, we study the benefits of TTT for TabPFN.
  arXiv  Detail & Related papers  (2025-03-14T20:06:37Z)
• Test-Time Training on Graphs with Large Language Models (LLMs) [68.375487369596]
  Test-Time Training (TTT) has been proposed as a promising approach to train Graph Neural Networks (GNNs) Inspired by the great annotation ability of Large Language Models (LLMs) on Text-Attributed Graphs (TAGs), we propose to enhance the test-time training on graphs with LLMs as annotators. A two-stage training strategy is designed to tailor the test-time model with the limited and noisy labels.
  arXiv  Detail & Related papers  (2024-04-21T08:20:02Z)
• Deja vu: Contrastive Historical Modeling with Prefix-tuning for Temporal Knowledge Graph Reasoning [16.408149489677154]
  ChapTER is a Contrastive historical modeling framework with prefix-tuning for TEmporal Reasoning. We evaluate ChapTER on four transductive and three few-shot inductive TKGR benchmarks.
  arXiv  Detail & Related papers  (2024-03-25T17:25:40Z)
• Test-Time Training on Video Streams [66.63237260332984]
  Prior work has established Test-Time Training (TTT) as a general framework to further improve a trained model at test time.<n>We extend TTT to the streaming setting, where multiple test instances arrive in temporal order.<n>Online TTT significantly outperforms the fixed-model baseline for four tasks, on three real-world datasets.
  arXiv  Detail & Related papers  (2023-07-11T05:17:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.