SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

• URL: http://arxiv.org/abs/2608.22354v2
• Date: Tue, 25 Aug 2026 03:15:23 GMT
• Title: SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models
• Abstract summary: Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
• Score: 53.86918766240095
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: \textbf{localized norm explosion atop a relatively sparse substrate}, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose \textbf{State Anomaly Neutralization (SANE)}, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3 \le α\le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $24{,}000\times$, SANE retains functional reasoning ($33.46$--$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($α\ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity--stability trade-off in state compression.

Related papers

• Constrained Online Learning with Noisy Constraint Values [55.29259818039367]
  Under common feasibility, our LEDGER algorithm attains $O(sqrt T)$ expected regret and $O(sqrtTlog(eT))$ expected budget violation.<n>Neither a Slater condition, independence between feedback channels, nor an absolute constraint-value bound is needed.
  arXiv  Detail & Related papers  (2026-09-07T01:38:41Z)
• Fast Weight Attention for Continual Learning [75.34672054079408]
  Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state.<n>We derive normalized first-order updates for squared-error regression and negative inner-product objectives.<n>We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization.
  arXiv  Detail & Related papers  (2026-08-27T22:55:11Z)
• Stateful CARS: Exact Cross-History Reuse for Policy-Constrained LLM Agents [0.9558392439655014]
  We study exact sampling from the model distribution conditioned on a hard stateful validator.<n>Stateful CARS freezes a bank of sound state--continuation schemas within each attempt.<n>Cross-history transfer helps only in an internal matched-key ablation.
  arXiv  Detail & Related papers  (2026-08-08T18:26:25Z)
• Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions [51.50375419691955]
  Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty.<n>We study how many samples are necessary and sufficient to learn an $varepsilon$-optimal robust policy under the average-reward criterion.
  arXiv  Detail & Related papers  (2026-08-06T19:49:48Z)
• Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory [7.4750926696730895]
  Naju factorizes the recurrent update, schematically $x_n = f_nodot x_n-1 + i_nodot(B_n u_n)$, into an explicit discrete pole.<n>Naju consistently combines strong long-range memory with competitive or superior performance.
  arXiv  Detail & Related papers  (2026-07-23T07:34:07Z)
• Is Spurious Correlation Removal Always Learnable? [56.28155520961125]
  Invariant learning can fail even when the structure is statistically identifiable.<n>We show a conditional computational barrier: under a black-box samplable supervised sparse recovery primitive, there exist emphamplable multi-dimensional environments.<n>Synthetic and real datasets illustrate the predicted gaps and transitions and motivate simple diversity diagnostics.
  arXiv  Detail & Related papers  (2026-06-11T05:49:43Z)
• When Does Dynamic Preconditioning Preserve the Polyak-Ruppert CLT? A Stabilization Threshold [12.805268849262243]
  Wald-type online inference remains valid for dynamically preconditioned averaged SGD whose stabilization rate exceeds the threshold.<n>We prove that within the class of rate hypotheses used in our upper bound, it cannot be weakened.
  arXiv  Detail & Related papers  (2026-04-26T02:14:19Z)
• Score-Repellent Monte Carlo: Toward Efficient Non-Markovian Sampler with Constant Memory in General State Spaces [48.37754141688874]
  We propose a framework that summarizes history by a running average of score evaluations in $Rd$, where $d$ is the dimension of the score and state representation.<n>This history is converted into a surrogate target through an exponential score tilt, indexed with $$ that represents the strength of repellence in controlling the magnitude of the history-based repulsion.<n>Experiments on continuous targets and discrete energy-based models demonstrate improved estimator variance and mode coverage, while retaining $O(d)$ memory usage and modest per-it overhead.
  arXiv  Detail & Related papers  (2026-04-24T18:39:50Z)
• Benchmarking the Utility of Privacy-Preserving Cox Regression Under Data-Driven Clipping Bounds: A Multi-Dataset Simulation Study [0.0]
  Differential privacy (DP) is a mathematical framework that guarantees individual privacy.<n>This study systematically evaluated the impact of DP mechanisms with data-driven clipping bounds on the Cox proportional hazards model.
  arXiv  Detail & Related papers  (2026-04-23T09:53:15Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Avoiding Premature Collapse: Adaptive Annealing for Entropy-Regularized Structural Inference [1.7523718031184992]
  We identify a fundamental mechanism for this failure: textbfPremature Mode Collapse.<n>We propose textbfEfficient Piecewise Hybrid Adaptive Stability Control (EPH-ASC), an adaptive scheduling algorithm that monitors the stability of the inference process.
  arXiv  Detail & Related papers  (2026-01-30T14:47:18Z)
• Spectral Sentinel: Scalable Byzantine-Robust Decentralized Federated Learning via Sketched Random Matrix Theory on Blockchain [0.0]
  Byzantine clients poison gradients under heterogeneous (Non-IID) data.<n>We propose Spectral Sentinel, a Byzantine detection and aggregation framework.<n>We implement the full system with blockchain integration on Polygon networks.
  arXiv  Detail & Related papers  (2025-12-14T09:43:03Z)
• Towards a Sharp Analysis of Offline Policy Learning for $f$-Divergence-Regularized Contextual Bandits [49.96531901205305]
  We analyze $f$-divergence-regularized offline policy learning.<n>For reverse Kullback-Leibler (KL) divergence, we give the first $tildeO(epsilon-1)$ sample complexity under single-policy concentrability.<n>We extend our analysis to dueling bandits, and we believe these results take a significant step toward a comprehensive understanding of $f$-divergence-regularized policy learning.
  arXiv  Detail & Related papers  (2025-02-09T22:14:45Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.