Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

• URL: http://arxiv.org/abs/2607.21000v1
• Date: Thu, 23 Jul 2026 07:34:07 GMT
• Title: Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
• Abstract summary: Naju factorizes the recurrent update, schematically $x_n = f_nodot x_n-1 + i_nodot(B_n u_n)$, into an explicit discrete pole.<n>Naju consistently combines strong long-range memory with competitive or superior performance.
• Score: 7.4750926696730895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well. Continuous-time-parameterized state-space models (SSMs) such as Mamba obtain their discrete recurrence by zero-order-hold discretization of a continuous-time system; we argue that this detour is unnecessary for memory tracking and parameterize the discrete transition directly. Naju (Native Adaptive Junction Unit) factorizes the recurrent update, schematically $x_n = f_n\odot x_{n-1} + i_n\odot(B_n u_n)$, into an explicit discrete pole (a learned forget gate $f_n$), an independent write gain $i_n$, and input-dependent write/read maps. Since the sigmoid pole satisfies $0<f_n<1$, each frozen local coordinate is Schur-stable by construction, and the full time-varying recurrence satisfies a fading-memory/BIBO bound under uniform boundedness assumptions, with no stability regularizer. We formalize the key structural limitation of coupled designs: any non-expansive complementary single-gate recurrence ties the effective retention $r$ and write gain $w$ through $|r|+w\le 1$, so near-complete retention forces weak writing; decoupling $f_n$ from $i_n$ removes this constraint. Empirically, Naju is the only evaluated model that remains strong on both retention and overwriting at 4x the training length. Beyond the diagnostic suite, we evaluate Naju on WikiText-103 language modeling, Long Range Arena, and multi-query associative recall. Across these settings, Naju consistently combines strong long-range memory with competitive or superior performance, outperforming the Mamba baselines in the principal comparisons while remaining competitive with the Transformer and preserving linear-time, linear-memory scaling.

Related papers

• Fast Weight Attention for Continual Learning [75.34672054079408]
  Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state.<n>We derive normalized first-order updates for squared-error regression and negative inner-product objectives.<n>We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization.
  arXiv  Detail & Related papers  (2026-08-27T22:55:11Z)
• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• Quantum Coordination Advantages in AI State-Tracking Tasks: Semantic Compilation and Latent Memory [5.649244092106778]
  We prove inference-time quantum coordination advantages for specified AI state-tracking tasks.<n>We map a finite one-way, streaming, or adaptive causal task into a semantic AI interface while preserving event order and access to past input.
  arXiv  Detail & Related papers  (2026-08-11T15:32:15Z)
• Raven: High-Recall Sequence Modeling with Sparse Memory Routing [88.47618436676748]
  We introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing.<n>Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade.
  arXiv  Detail & Related papers  (2026-07-28T07:04:32Z)
• LongSpike: Fractional Order Spiking State Space Models for Efficient Long Sequence Learning [53.855185131868446]
  Spiking Neural Networks (SNNs) are well-regarded for their biological plausibility and energy efficiency in processing sequential data.<n>In this work, we propose LongSpike, a novel SNN framework that integrates fractional-order State-Space Modeling.<n>By extending traditional integer-order SSMs to the fractional-calculus regime, LongSpike enables the hierarchical integration of neuronal dynamics with long-memory kernels.
  arXiv  Detail & Related papers  (2026-06-11T04:54:07Z)
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference [58.46323651532913]
  We study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights.<n>During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model.<n>We show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.
  arXiv  Detail & Related papers  (2026-05-25T17:55:39Z)
• Variational Linear Attention: Stable Associative Memory for Long-Context Transformers [0.6117371161379209]
  Linear attention reduces the quadratic cost of softmax attention to $mathcalO(T)$, but its memory state grows as $mathcalO(T)$ in Frobenius norm.<n>We introduce textbfVariational Linear Attention (VLA), which reframes the memory update as an online regularised least-squares problem.
  arXiv  Detail & Related papers  (2026-05-11T20:03:12Z)
• TIDES: Implicit Time-Awareness in Selective State Space Models [0.0]
  Continuous-time SSMs handle irregular timestamps, but their dynamics remain linear time-invariant (LTI)<n>We propose textbfTIDES, a selective SSM variant that reconciles selective and continuous architectures.<n> TIDES sets the new state-of-the-art average rank on UEA time-series classification and the Physiome-ODE regression benchmark.
  arXiv  Detail & Related papers  (2026-05-10T20:34:06Z)
• Score-Repellent Monte Carlo: Toward Efficient Non-Markovian Sampler with Constant Memory in General State Spaces [48.37754141688874]
  We propose a framework that summarizes history by a running average of score evaluations in $Rd$, where $d$ is the dimension of the score and state representation.<n>This history is converted into a surrogate target through an exponential score tilt, indexed with $$ that represents the strength of repellence in controlling the magnitude of the history-based repulsion.<n>Experiments on continuous targets and discrete energy-based models demonstrate improved estimator variance and mode coverage, while retaining $O(d)$ memory usage and modest per-it overhead.
  arXiv  Detail & Related papers  (2026-04-24T18:39:50Z)
• Sessa: Selective State Space Attention [0.0]
  We propose Sessa, a decoder that places attention inside a recurrent feedback path.<n>We prove that Sessa admits power-law memory tails $O(ell-)$ for $0 1$, with slower decay than in the corresponding Transformer and Mamba-style baselines.
  arXiv  Detail & Related papers  (2026-04-20T17:59:08Z)
• Quantifying Memory Use in Reinforcement Learning with Temporal Range [51.98491034847041]
  Temporal Range is a model-agnostic metric that treats first-order sensitivities of multiple vector outputs across a temporal window to the input sequence as a temporal influence profile.<n>We also report Temporal Range for a compact Long Expressive Memory (LEM) policy trained on the task, using it as a proxy readout of task-level memory.
  arXiv  Detail & Related papers  (2025-12-05T22:58:09Z)
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression [53.48692193399171]
  Gated KalmaNet (GKA) is a layer that reduces the gap by accounting for the full past when predicting the next token.<n>We solve an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.<n>On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
  arXiv  Detail & Related papers  (2025-11-26T03:26:37Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• $\infty$-former: Infinite Memory Transformer [5.262376184578169]
  Transformers struggle when attending to long contexts, since the amount of computation grows with the context length. We propose the $infty$-former, which extends the vanilla transformer with an unbounded long-term memory.
  arXiv  Detail & Related papers  (2021-09-01T10:51:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.