HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models

• URL: http://arxiv.org/abs/2602.21340v1
• Date: Tue, 24 Feb 2026 20:16:54 GMT
• Title: HiPPO Zoo: Explicit Memory Mechanisms for Interpretable State Space Models
• Abstract summary: We revisit the HiPPO framework with the goal of making representations of history largely implicit.<n>We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo"
• Score: 1.4018975578160686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Representing the past in a compressed, efficient, and informative manner is a central problem for systems trained on sequential data. The HiPPO framework, originally proposed by Gu & Dao et al., provides a principled approach to sequential compression by projecting signals onto orthogonal polynomial (OP) bases via structured linear ordinary differential equations. Subsequent works have embedded these dynamics in state space models (SSMs), where HiPPO structure serves as an initialization. Nonlinear successors of these SSM methods such as Mamba are state-of-the-art for many tasks with long-range dependencies, but the mechanisms by which they represent and prioritize history remain largely implicit. In this work, we revisit the HiPPO framework with the goal of making these mechanisms explicit. We show how polynomial representations of history can be extended to support capabilities of modern SSMs such as adaptive allocation of memory and associative memory while retaining direct interpretability in the OP basis. We introduce a unified framework comprising five such extensions, which we collectively refer to as a "HiPPO zoo." Each extension exposes a specific modeling capability through an explicit, interpretable modification of the HiPPO framework. The resulting models adapt their memory online and train in streaming settings with efficient updates. We illustrate the behaviors and modeling advantages of these extensions through a range of synthetic sequence modeling tasks, demonstrating that capabilities typically associated with modern SSMs can be realized through explicit, interpretable polynomial memory structures.

Related papers

• Learning Long Range Spatio-Temporal Representations over Continuous Time Dynamic Graphs with State Space Models [22.610672356493378]
  Continuous-time dynamic graphs (CTDGs) provide a richer framework to capture fine-grained temporal patterns in evolving data.<n>We derive a parameter-efficient state-space modeling framework for continuous-time dynamic graphs.<n>CTDG-SSM achieves large performance gains on datasets that require long range temporal (LRT) and spatial reasoning.
  arXiv  Detail & Related papers  (2026-06-03T09:54:35Z)
• Flash PD-SSM: Memory-Optimized Structured Sparse State-Space Models [62.362971723881095]
  State-space models (SSMs) face a fundamental trade-off between efficiency and expressivity.<n>We propose Flash PD-SSM, a novel SSM that achieves comparable throughput to widely-used structured SSMs with significantly better expressivity guarantees.<n>We show that Flash PD-SSM defines a new state-of-the-art (SoTA) accuracy among competing SSM methods.
  arXiv  Detail & Related papers  (2026-05-18T22:06:33Z)
• Lag Operator SSMs: A Geometric Framework for Structured State Space Modeling [3.3864018929063477]
  We introduce a framework for constructing discrete-time Structured State Space Models (SSMs) that is both flexible and modular.<n>Our approach is based on a novel lag operator, which geometrically derives the discrete-time recurrence by measuring how the system's basis functions "slide"
  arXiv  Detail & Related papers  (2025-12-22T02:25:26Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• SaFARi: State-Space Models for Frame-Agnostic Representation [22.697360024988484]
  State-Space Models (SSMs) have re-emerged as a powerful tool for online function, and as the backbone of machine learning models for long-range dependent data.<n>We present a method for building an SSM with any frame or basis, rather than being restricted tognostics.<n>This framework encompasses the approach known as HiPPO, but also permits an infinite diversity of other possible "species" within the SSM architecture.
  arXiv  Detail & Related papers  (2025-05-13T21:39:40Z)
• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• Recurrent Memory for Online Interdomain Gaussian Processes [14.986242786275183]
  We propose a novel online Gaussian process (GP) model that is capable of capturing long-term memory in sequential data in an online learning setting.<n>Our model, Online HiPPO Sparse Variational Gaussian Process (OHSVGP), leverages the HiPPO framework, which is popularized in the RNN domain due to its long-range memory modeling capabilities.
  arXiv  Detail & Related papers  (2025-02-12T19:18:50Z)
• Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms [0.6718184400443239]
  We propose an advanced architecture that mitigates challenges by decomposing A-multiplications into multiple groups. Inspired by the "attention sink" phenomenon identified in streaming language models, we incorporate a similar mechanism to enhance the stability and performance of our model.
  arXiv  Detail & Related papers  (2024-08-01T02:49:58Z)
• Longhorn: State Space Models are Amortized Online Learners [51.10124201221601]
  State-space models (SSMs) offer linear decoding efficiency while maintaining parallelism during training. In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. We introduce a novel deep SSM architecture, Longhorn, whose update resembles the closed-form solution for solving the online associative recall problem.
  arXiv  Detail & Related papers  (2024-07-19T11:12:08Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)
• PAC Reinforcement Learning for Predictive State Representations [60.00237613646686]
  We study online Reinforcement Learning (RL) in partially observable dynamical systems. We focus on the Predictive State Representations (PSRs) model, which is an expressive model that captures other well-known models. We develop a novel model-based algorithm for PSRs that can learn a near optimal policy in sample complexity scalingly.
  arXiv  Detail & Related papers  (2022-07-12T17:57:17Z)
• S2RMs: Spatially Structured Recurrent Modules [105.0377129434636]
  We take a step towards exploiting dynamic structure that are capable of simultaneously exploiting both modular andtemporal structures. We find our models to be robust to the number of available views and better capable of generalization to novel tasks without additional training.
  arXiv  Detail & Related papers  (2020-07-13T17:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.