Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection

• URL: http://arxiv.org/abs/2608.02560v1
• Date: Mon, 03 Aug 2026 17:43:36 GMT
• Title: Structured Memory for Edge Language Models: Persistent Context and Corpus Retrieval via O(1) SSM State Injection
• Abstract summary: State-Space Models (SSMs) avoid prefill costs by construction.<n>Pre-encodes document corpora offline as SSM hidden states.<n>Prefill latency is reduced from $sim$27 s to $$6 ms on edge hardware.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented generation (RAG) imposes a prefill cost proportional to retrieved context length, and -- with Transformer backbones -- a KV-cache that grows with each generated token. State-Space Models (SSMs) avoid the second cost by construction; we eliminate the first, collapsing prefill from $O(L_{context})$ to $O(1)$ per query. We introduce PRECOG (Pre-Computed Context Injection), a retrieval mechanism that exploits a property unique to SSMs: the fixed-size, position-agnostic recurrent hidden state is a complete summary of everything the model has read. PRECOG pre-encodes document corpora offline as SSM hidden states and injects the best-matching state directly at query time, bypassing in-context re-ingestion entirely. The same state-injection mechanism enables SMC (Structured Memory Consolidation): a hierarchical persistent memory with cognitive-domain clustering, an adjustable fidelity-vs-storage dial, and $O(1)$ session initialization, which consolidates short-term episodic states into long-term semantic memory and fuses both with retrieved corpus states at query time. We demonstrate the system on TENNs-LLM, a 1.2B-parameter gated-SSM language model with a 192 KB hidden state. PRECOG matches in-context RAG answer quality, reducing prefill latency from $\sim$27 s to $<$6 ms on edge hardware -- a $\sim$4500$\times$ speedup that crosses the threshold from unusable to interactive. The mechanism is architecturally impossible for Transformer KV-caches, which are position-entangled and grow linearly with context length.

Related papers

• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems [5.321970297768025]
  Existing memory architectures treat all memories as equally persistent and systematically contaminate retrieved context with outdated facts.<n>We show that per-memory, type-conditioned temporal decay, can be operationalized as an auto-classified coefficient $_i$ in an external LLM-agent memory store.<n>We introduce the Temporal Generalization Test (TGT), a benchmark with held-out retention intervals and a Generalization Gap (GenGap) metric.
  arXiv  Detail & Related papers  (2026-08-05T12:12:44Z)
• Scaling Self-Evolving Agents via Parametric Memory [69.96398842169002]
  Existing memory-augmented LLM agents store past experience exclusively in prompt space.<n>We introduce textttTMEM, a self-evolving parametric memory framework.<n>textttTMEM consistently outperforms summary-based and retrieval-based baselines across different model scales.
  arXiv  Detail & Related papers  (2026-06-03T07:18:31Z)
• DMF: A Deterministic Memory Framework for Conversational AI Agents [0.0]
  We present the Deterministic Memory Framework (DMF), a CPU-first approach that replaces generative memory compression with a fully deterministic pipeline.<n>DMF assigns each conversational interaction a Survival Score $$ computed from deterministic content signals, conversational cues, and structured provenance, combined through a logistic projection.<n>Experiments are conducted on a purpose-built benchmark using the LoCoMo and LongMemEval datasets.
  arXiv  Detail & Related papers  (2026-06-02T10:41:28Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators [45.88028371034407]
  KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA)<n>We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA)
  arXiv  Detail & Related papers  (2026-05-07T22:26:27Z)
• Sparse Prefix Caching for Hybrid and Recurrent LLM Serving [0.7284556903703034]
  We formalize sparse prefix caching as checkpoint placement under a distribution over overlap depths.<n>For use cases where requests share a non-trivial prefix, we show that our method consistently improves the frontier traced by standard asymmetry on real-world data.<n>It preserves exact outputs, does not change the recurrent computation itself or require new recurrent update kernels.
  arXiv  Detail & Related papers  (2026-04-17T09:24:58Z)
• Persistent Memory Through Triple-Loop Consolidation in a Non-Gradient Dissipative Cognitive Architecture [0.0]
  We introduce Deep Memory, a non-gradient persistent memory mechanism operating through a triple-loop consolidation cycle.<n>We demonstrate that discrete expert routing via Mixture-of-Experts (MoE) gating is a causal prerequisite for DM.<n>These results establish DM as a falsifiable mechanism for persistent memory in non-gradient cognitive systems, with functional parallels to hippocampal consolidation.
  arXiv  Detail & Related papers  (2026-03-28T08:29:48Z)
• The Bureaucracy of Speed: Structural Equivalence Between Memory Consistency Models and Multi-Agent Authorization Revocation [0.0]
  We develop a Capability Coherence System for Identity and Access Management.<n>A safety theorem bounds unauthorized operations for the execution-count Release Consistency-directed Coherence strategy.<n> Tick-based discrete event simulation confirms per-capability safety guarantee.
  arXiv  Detail & Related papers  (2026-03-10T16:37:02Z)
• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space [56.37266873329401]
  Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density.<n>We propose $textbfDynamic Large Concept Models (DLCM)$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts from tokens to a compressed concept space where reasoning is more efficient.
  arXiv  Detail & Related papers  (2025-12-31T04:19:33Z)
• Structured Sparse Transition Matrices to Enable State Tracking in State-Space Models [68.31088463716269]
  We propose a structured sparse parametrization of transition matrices in state-space models (SSMs)<n>Our method, PD-SSM, parametrizes the transition matrix as the product of a column one-hot matrix ($P$) and a complex-valued diagonal matrix ($D$)<n>The model significantly outperforms a wide collection of modern SSM variants on various FSA state tracking tasks.
  arXiv  Detail & Related papers  (2025-09-26T12:46:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.