StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation

• URL: http://arxiv.org/abs/2603.23571v1
• Date: Tue, 24 Mar 2026 11:04:50 GMT
• Title: StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigation
• Abstract summary: We introduce StateLinFormer, a linear-attention navigation model trained with a stateful memory mechanism.<n>This training paradigm effectively approximates learning on infinitely long sequences, enabling the model to achieve long-horizon memory retention.
• Score: 14.66409147430344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective navigation intelligence relies on long-term memory to support both immediate generalization and sustained adaptation. However, existing approaches face a dilemma: modular systems rely on explicit mapping but lack flexibility, while Transformer-based end-to-end models are constrained by fixed context windows, limiting persistent memory across extended interactions. We introduce StateLinFormer, a linear-attention navigation model trained with a stateful memory mechanism that preserves recurrent memory states across consecutive training segments instead of reinitializing them at each batch boundary. This training paradigm effectively approximates learning on infinitely long sequences, enabling the model to achieve long-horizon memory retention. Experiments across both MAZE and ProcTHOR environments demonstrate that StateLinFormer significantly outperforms its stateless linear-attention counterpart and standard Transformer baselines with fixed context windows. Notably, as interaction length increases, persistent stateful training substantially improves context-dependent adaptation, suggesting an enhancement in the model's In-Context Learning (ICL) capabilities for navigation tasks.

Related papers

• Extending LLM Context via Associative Recurrent Memory [31.93460198288273]
  We investigate the Associative Recurrent Memory Transformer (ARMT) as a practical approach for enabling long-context processing in large language models (LLMs)<n>Second, we propose a comprehensive training recipe for ARMT-based context extension, combining continued pre-training, synthetic long-context data generation, curriculum learning, and selective integration of associative memory into chosen model layers.<n>Third, we present an extensive experimental study demonstrating that ARMT-augmented models: (i) process inputs well beyond their original context limits without degrading performance relative to in-limit baselines; (ii) generalize more effectively
  arXiv  Detail & Related papers  (2026-07-13T14:37:24Z)
• SAM: State-Adaptive Memory for Long-Horizon Reasoning Agent [51.274445160155864]
  Long-horizon agentic reasoning requires large language models to act over long interaction histories containing thoughts, tool calls, observations, and partial conclusions.<n>Existing approaches address this difficulty by truncating the interaction history, compressing it into shorter surrogates, or retrieving selected parts of it for reuse.<n>We propose State-Adaptive Memory, a standalone framework that consolidates ongoing interaction into compact memory cues while preserving raw trajectory pages for intent-driven recall.
  arXiv  Detail & Related papers  (2026-05-23T08:37:16Z)
• Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory [53.8372503928207]
  Large language model (LLM) agents require long-term user memory for consistent personalization.<n>Existing memory systems mainly rely on static, hand-crafted update rules.<n>We introduce MemCoE, a cognition-inspired two-stage optimization framework that learns how memory should be organized and what information to update.
  arXiv  Detail & Related papers  (2026-05-01T14:45:20Z)
• Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns [0.16921396880325779]
  We introduce TRC$2$ (Thalamically Routed Cortical Columns), a decoder-only backbone that addresses continual learning at the architectural level.<n>The resulting block is sparse and chunk-parallel, enabling efficient training and inference while preserving clean ablations of each subsystem.
  arXiv  Detail & Related papers  (2026-02-25T23:38:16Z)
• Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learning [53.72709564555407]
  Memo is a transformer-based architecture and training recipe for reinforcement learning.<n>It incorporates the creation and retrieval of memory by interleaving periodic summarization tokens with the inputs of a model during training.<n>We demonstrate Memo's effectiveness on a gridworld meta-RL benchmark and a multi-object navigation task in photo-realistic indoor settings.
  arXiv  Detail & Related papers  (2025-10-22T16:24:47Z)
• Lizard: An Efficient Linearization Framework for Large Language Models [113.87302474262798]
  We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures.<n>Lizard addresses these limitations by introducing a subquadratic attention mechanism that closely approximates softmax attention while preserving model quality.<n>We show that Lizard achieves near-lossless recovery of its teacher model's performance, significantly outperforming previous methods by up to 9.4 - 24.5 points on the 5-shot MMLU benchmark.
  arXiv  Detail & Related papers  (2025-07-11T21:19:18Z)
• Bisecle: Binding and Separation in Continual Learning for Video Language Understanding [11.710573955384511]
  We propose Bisecle for video-language continual learning, inspired by the rapid Binding and pattern separation mechanisms in the hippocampus.<n>Bisecle mitigates forgetting and enhances cross-task generalization on several VideoQA benchmarks.
  arXiv  Detail & Related papers  (2025-07-01T06:28:57Z)
• Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning [64.93848182403116]
  Current deep-learning memory models struggle in reinforcement learning environments that are partially observable and long-term. We introduce the Stable Hadamard Memory, a novel memory model for reinforcement learning agents. Our approach significantly outperforms state-of-the-art memory-based methods on challenging partially observable benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T03:50:17Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters [65.15700861265432]
  We present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters. To preserve the zero-shot recognition capability of vision-language models, we introduce a Distribution Discriminative Auto-Selector.
  arXiv  Detail & Related papers  (2024-03-18T08:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.