Self-Guided Test-Time Training for Long-Context LLMs

• URL: http://arxiv.org/abs/2607.09415v1
• Date: Fri, 10 Jul 2026 13:45:56 GMT
• Title: Self-Guided Test-Time Training for Long-Context LLMs
• Abstract summary: Long-context processing is increasingly important for large language models.<n>As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question.<n>A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation.<n>We propose Self-Guided TTT (S-TTT), which identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans.
• Score: 16.160114043456215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-context processing has become increasingly important for large language models (LLMs), but simply extending the context window does not guarantee effective utilization of long inputs. As input length grows, accuracy often degrades, indicating that models still struggle to identify and use the evidence most relevant to a question. A promising way to improve long-context utilization is test-time training (TTT), which treats the test context as a training example for instance-specific parameter adaptation. However, applying TTT to the entire long context is prohibitively expensive, while adapting on randomly sampled spans introduces severe noise. Because most spans in a long context are irrelevant to the specific question, training on them may even degrade the base model's performance. Our preliminary study shows that TTT is highly sensitive to training-span quality: on LongBench-v2, TTT on randomly sampled spans hurts performance, whereas TTT on oracle spans substantially improves it. Motivated by this, we propose a simple method, Self-Guided TTT (S-TTT): before adaptation, the model identifies the evidence spans it should learn from, and the standard language-modeling training objective is applied only to those selected spans. On two challenging long-context reasoning benchmarks, LongBench-v2 and LongBench-Pro, S-TTT improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.

Related papers

• Learning What to Remember: Test-Time Training via Context Distillation [72.10421333095852]
  Test-time training (TTT) is an appealing approach that performs online parameter updates for long-context modeling.<n>We propose textbfTest-textbfTime textbfContext textbfDistillation (TTCD), a TTT framework that introduces a self-supervised objective for allocating limited memory capacity for future use.
  arXiv  Detail & Related papers  (2026-08-03T04:06:06Z)
• EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering [61.89411578705886]
  We propose Evidence-Aligned SElective Test-Time Training (EASE-TTT) for long-context question answering.<n>EASE-TTT converts selected evidence chunks into a soft attention supervision target over their token positions.<n>It achieves the strongest macro-average performance among full-context inference, retrieval-only baselines, and qTTT.
  arXiv  Detail & Related papers  (2026-06-05T04:49:37Z)
• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation [86.62036852878354]
  Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding.<n>We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness.<n> Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks.
  arXiv  Detail & Related papers  (2026-04-15T18:46:35Z)
• End-to-End Test-Time Training for Long Context [98.3930777591529]
  We formulate long-context language modeling as a problem in continual learning rather than architecture design.<n>Our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights.<n>Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning)
  arXiv  Detail & Related papers  (2025-12-29T18:30:14Z)
• How to Train Long-Context Language Models (Effectively) [75.5418485597276]
  We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information.<n>We find that code repositories and books are excellent sources of long data, but it is crucial to combine them with high-quality short-context data.<n>Our final model, ProLong-8B, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K.
  arXiv  Detail & Related papers  (2024-10-03T16:46:52Z)
• Effective Long-Context Scaling of Foundation Models [90.57254298730923]
  We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2.
  arXiv  Detail & Related papers  (2023-09-27T21:41:49Z)
• Improved Test-Time Adaptation for Domain Generalization [48.239665441875374]
  Test-time training (TTT) adapts the learned model with test data. This work addresses two main factors: selecting an appropriate auxiliary TTT task for updating and identifying reliable parameters to update during the test phase. We introduce additional adaptive parameters for the trained model, and we suggest only updating the adaptive parameters during the test phase.
  arXiv  Detail & Related papers  (2023-04-10T10:12:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.