Absorber LLM: Harnessing Causal Synchronization for Test-Time Training

• URL: http://arxiv.org/abs/2604.20915v1
• Date: Wed, 22 Apr 2026 02:58:26 GMT
• Title: Absorber LLM: Harnessing Causal Synchronization for Test-Time Training
• Abstract summary: Transformers suffer from a high computational cost that grows with sequence length for self-attention.<n> Constant-memory alternatives such as RNNs compress history into states with fixed size and thus lose long-tail dependencies.<n>We propose Absorber LLM, which formulates long-context retention as a self-supervised causal synchronization.
• Score: 11.369402753246396
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers suffer from a high computational cost that grows with sequence length for self-attention, making inference in long streams prohibited by memory consumption. Constant-memory alternatives such as RNNs and SSMs compress history into states with fixed size and thus lose long-tail dependencies, while methods that memorize contexts into parameters, such as Test-Time Training (TTT), are prone to overfitting token-level projection and fail to preserve the causal effect of context in pretrained LLMs. We propose Absorber LLM, which formulates long-context retention as a self-supervised causal synchronization: after absorbing historical contexts into parameters, a contextless model should match the original model with full context on future generations. We optimize this objective by synchronizing internal behaviors of the updated model with the original one, ensuring context absorption and generalization. Experiments on long-context and streaming benchmarks show that Absorber LLM reduces inference memory and improves accuracy over prior parameter-as-memory baselines.

Related papers

• Extending LLM Context via Associative Recurrent Memory [31.93460198288273]
  We investigate the Associative Recurrent Memory Transformer (ARMT) as a practical approach for enabling long-context processing in large language models (LLMs)<n>Second, we propose a comprehensive training recipe for ARMT-based context extension, combining continued pre-training, synthetic long-context data generation, curriculum learning, and selective integration of associative memory into chosen model layers.<n>Third, we present an extensive experimental study demonstrating that ARMT-augmented models: (i) process inputs well beyond their original context limits without degrading performance relative to in-limit baselines; (ii) generalize more effectively
  arXiv  Detail & Related papers  (2026-07-13T14:37:24Z)
• Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling [65.65882819517455]
  We propose a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss.<n> Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths.
  arXiv  Detail & Related papers  (2026-07-03T05:39:00Z)
• MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models [64.2820121997882]
  MIITA is a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage.<n>It stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues.<n>Experiments show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.
  arXiv  Detail & Related papers  (2026-05-20T03:03:04Z)
• Developing Adaptive Context Compression Techniques for Large Language Models (LLMs) in Long-Running Interactions [0.0]
  Large Language Models (LLMs) often experience performance degradation during long-running interactions due to increasing context length, memory saturation, and computational overhead.<n>This paper presents an adaptive context compression framework that integrates importance-aware memory selection, coherence-sensitive filtering, and dynamic budget allocation to retain essential conversational information while controlling context growth.
  arXiv  Detail & Related papers  (2026-03-31T02:57:48Z)
• MemDLM: Memory-Enhanced DLM Training [56.40248490616793]
  Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models.<n>They suffer from a notable train-inference mismatch.<n>We propose MemDLM (Memory-Enhanced DLM), which embeds a simulated denoising process into training.
  arXiv  Detail & Related papers  (2026-03-23T17:39:56Z)
• TS-Haystack: A Multi-Scale Retrieval Benchmark for Time Series Language Models [4.387988928531881]
  Time Series Language Models (TSLMs) are emerging as unified models for reasoning over continuous signals in natural language.<n>Existing models are typically trained and evaluated on short sequences, while real-world time-series sensor streams can span millions of datapoints.<n>We introduce TS-Haystack, a long-context temporal retrieval benchmark comprising ten task types across four categories.
  arXiv  Detail & Related papers  (2026-02-15T15:50:02Z)
• Recurrent Preference Memory for Efficient Long-Sequence Generative Recommendation [27.325586037888]
  We introduce Rec2PM, a framework that compresses long user interaction histories into compact Preference Memory tokens.<n>Experiments show that Rec2PM significantly reduces inference latency and memory footprint while achieving superior accuracy compared to full-sequence models.
  arXiv  Detail & Related papers  (2026-02-12T05:51:52Z)
• TS-Memory: Plug-and-Play Memory for Time Series Foundation Models [63.21390142212087]
  Time Series Foundation Models (TSFMs) achieve strong zero-shot forecasting through large-scale pre-training.<n>Existing solutions face a trade-off: Parametric Adaptation can cause catastrophic forgetting, while Non-Parametric Retrieval improves forecasts but incurs high latency due to datastore search.<n>We propose Parametric Memory Distillation and implement it as TS-Memory, a lightweight memory adapter that augments frozen TSFMs.
  arXiv  Detail & Related papers  (2026-02-12T04:16:19Z)
• Latent Context Compilation: Distilling Long Context into Compact Portable Memory [13.768393657432027]
  We propose Latent Context Compilation, a framework that shifts context processing from adaptation to compilation.<n>By utilizing a disposable LoRA module as a compiler, we distill long contexts into compact buffer tokens.<n> Experiments with Llama-3.1-8B demonstrate that Latent Context Compilation preserves fine-grained details and reasoning capabilities.
  arXiv  Detail & Related papers  (2026-01-31T08:38:07Z)
• Mixture of Contexts for Long Video Generation [72.96361488755986]
  We recast long-context video generation as an internal information retrieval task.<n>We propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.<n>As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content.
  arXiv  Detail & Related papers  (2025-08-28T17:57:55Z)
• Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache [67.47789629197857]
  We propose a training-free framework that exploits the heterogeneous roles of transformer head dimensions.<n>By projecting the long-context-insensitive dimensions onto Fourier bases, FourierAttention approximates their temporal evolution with fixed-length spectral coefficients.<n>We show that FourierAttention achieves the best long-context accuracy on LongBench and Needle-In-A-Haystack.
  arXiv  Detail & Related papers  (2025-06-13T15:35:54Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.