Fast Weight Attention for Continual Learning

• URL: http://arxiv.org/abs/2608.27763v1
• Date: Thu, 27 Aug 2026 22:55:11 GMT
• Title: Fast Weight Attention for Continual Learning
• Abstract summary: Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state.<n>We derive normalized first-order updates for squared-error regression and negative inner-product objectives.<n>We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization.
• Score: 75.34672054079408
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step $t$ is the prefix-aligned pair $(\mathbf{x}_t,\mathbf{y}_t)=(φ(\mathbf{k}_{t-1}),\mathbf{v}_t)$. The common same-step association $(φ(\mathbf{k}_t),\mathbf{v}_t)$ remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.

Related papers

• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• High-Dimensional Nonparametric Change-Point Detection via Low-Rank Degree-Three Density Projection [2.1991623402625606]
  Distributional changes can be invisible to means and covariances yet appear in skewness, asymmetric interactions, or other third-order structure.<n>We develop a nonparametric change-point method that retains every degree-at-most-three coefficient of a density while avoiding direct density estimation.
  arXiv  Detail & Related papers  (2026-08-16T01:21:55Z)
• Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory [7.4750926696730895]
  Naju factorizes the recurrent update, schematically $x_n = f_nodot x_n-1 + i_nodot(B_n u_n)$, into an explicit discrete pole.<n>Naju consistently combines strong long-range memory with competitive or superior performance.
  arXiv  Detail & Related papers  (2026-07-23T07:34:07Z)
• Scaling Laws for Behavioral Foundation Models over User Event Sequences [2.924581427482972]
  We study a common two-part behavioral-model architecture, a feature-based event embedder, and a decoder-only transformer.<n>Across roughly 600 runs on real interaction data, spanning $1015$-$1019$ training FLOPs, we vary four deployment-relevant axes.<n> Compute-optimal training is data-heavy relative to text at low compute, but its $D/N$ ratio moves toward the Chinchilla as compute increases.
  arXiv  Detail & Related papers  (2026-06-03T15:59:25Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models [27.658955798426323]
  AdamW has become one of the most effective generalizations for training large-scale models.<n>We propose the first underlineFederated underlineAdamW algorithm, called textttFedAdamW, for training and fine-tuning various large models.
  arXiv  Detail & Related papers  (2025-10-31T14:04:43Z)
• Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesis [55.561961365113554]
  3D Gaussian Splatting (3DGS) has demonstrated remarkable effectiveness in novel view synthesis (NVS)<n>In this paper, we introduce Self-Ensembling Gaussian Splatting (SE-GS)<n>We achieve self-ensembling by incorporating an uncertainty-aware perturbation strategy during training.<n> Experimental results on the LLFF, Mip-NeRF360, DTU, and MVImgNet datasets demonstrate that our approach enhances NVS quality under few-shot training conditions.
  arXiv  Detail & Related papers  (2024-10-31T18:43:48Z)
• High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation [89.21686761957383]
  We study the first gradient descent step on the first-layer parameters $boldsymbolW$ in a two-layer network. Our results demonstrate that even one step can lead to a considerable advantage over random features.
  arXiv  Detail & Related papers  (2022-05-03T12:09:59Z)
• Modeling Label Correlations for Second-Order Semantic Dependency Parsing with Mean-Field Inference [34.75002236767817]
  Second-order semantic parsing with end-to-end mean-field inference has been shown good performance. In this work we aim to improve this method by modeling label correlations between adjacent arcs. To tackle this computational challenge, we leverage tensor decomposition techniques.
  arXiv  Detail & Related papers  (2022-04-07T17:40:08Z)
• On Submodular Contextual Bandits [92.45432756301231]
  We consider the problem of contextual bandits where actions are subsets of a ground set and mean rewards are modeled by an unknown monotone submodular function. We show that our algorithm efficiently randomizes around local optima of estimated functions according to the Inverse Gap Weighting strategy.
  arXiv  Detail & Related papers  (2021-12-03T21:42:33Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.