WaLRUS: Wavelets for Long-range Representation Using SSMs

• URL: http://arxiv.org/abs/2505.12161v1
• Date: Sat, 17 May 2025 22:41:24 GMT
• Title: WaLRUS: Wavelets for Long-range Representation Using SSMs
• Authors: Hossein Babaei, Mel White, Sina Alemohammad, Richard G. Baraniuk,
• Abstract summary: State-Space Models (SSMs) have proven to be powerful tools for modeling long-range dependencies in sequential data.<n>We introduce WaLRUS, a new implementation of SaFARi built from Daubechies wavelets.
• Score: 22.697360024988484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-Space Models (SSMs) have proven to be powerful tools for modeling long-range dependencies in sequential data. While the recent method known as HiPPO has demonstrated strong performance, and formed the basis for machine learning models S4 and Mamba, it remains limited by its reliance on closed-form solutions for a few specific, well-behaved bases. The SaFARi framework generalized this approach, enabling the construction of SSMs from arbitrary frames, including non-orthogonal and redundant ones, thus allowing an infinite diversity of possible "species" within the SSM family. In this paper, we introduce WaLRUS (Wavelets for Long-range Representation Using SSMs), a new implementation of SaFARi built from Daubechies wavelets.

Related papers

• W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling [23.453158933852357]
  State Space Models (SSMs) have emerged as powerful components for sequence modeling.<n>We introduce a new variant, W4S4 (WaLRUS for S4), a new class of SSMs constructed from redundant wavelet frames.<n>We show that WaLRUS retains information over long horizons significantly better than HiPPO-based SSMs.
  arXiv  Detail & Related papers  (2025-06-09T16:33:29Z)
• Long-Context State-Space Video World Models [66.28743632951218]
  We propose a novel architecture leveraging state-space models (SSMs) to extend temporal memory without compromising computational efficiency.<n>Central to our design is a block-wise SSM scanning scheme, which strategically trades off spatial consistency for extended temporal memory.<n>Experiments on Memory Maze and Minecraft datasets demonstrate that our approach surpasses baselines in preserving long-range memory.
  arXiv  Detail & Related papers  (2025-05-26T16:12:41Z)
• Learning to Dissipate Energy in Oscillatory State-Space Models [55.09730499143998]
  State-space models (SSMs) are a class of networks for sequence learning.<n>We show that D-LinOSS consistently outperforms previous LinOSS methods on long-range learning tasks.
  arXiv  Detail & Related papers  (2025-05-17T23:15:17Z)
• SaFARi: State-Space Models for Frame-Agnostic Representation [22.697360024988484]
  State-Space Models (SSMs) have re-emerged as a powerful tool for online function, and as the backbone of machine learning models for long-range dependent data.<n>We present a method for building an SSM with any frame or basis, rather than being restricted tognostics.<n>This framework encompasses the approach known as HiPPO, but also permits an infinite diversity of other possible "species" within the SSM architecture.
  arXiv  Detail & Related papers  (2025-05-13T21:39:40Z)
• Enhancing Memory and Imagination Consistency in Diffusion-based World Models via Linear-Time Sequence Modeling [8.250616459360684]
  We introduce EDELINE, a novel framework that integrates diffusion models with linear-time state space models to enhance memory retention and temporal consistency.<n>Our results across multiple benchmarks demonstrate EDELINE's superiority and robustness over prior baselines in long-horizon tasks.
  arXiv  Detail & Related papers  (2025-02-01T15:49:59Z)
• Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning [54.19222454702032]
  Continual Learning aims to equip AI models with the ability to learn a sequence of tasks over time, without forgetting previously learned knowledge.<n>State Space Models (SSMs) have achieved notable success in computer vision.<n>We introduce Mamba-CL, a framework that continuously fine-tunes the core SSMs of the large-scale Mamba foundation model.
  arXiv  Detail & Related papers  (2024-11-23T06:36:16Z)
• Provable Benefits of Complex Parameterizations for Structured State Space Models [51.90574950170374]
  Structured state space models (SSMs) are linear dynamical systems adhering to a specified structure. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. This paper takes a step towards explaining the benefits of complex parameterizations for SSMs by establishing formal gaps between real and complex diagonal SSMs.
  arXiv  Detail & Related papers  (2024-10-17T22:35:50Z)
• Parameter-Efficient Fine-Tuning of State Space Models [10.817729275974829]
  Deep State Space Models (SSMs) have become powerful tools for language modeling, offering high performance and linear scalability with sequence length.<n>This paper investigates the application of parameter-efficient fine-tuning (PEFT) methods to SSM-based models.<n>We propose Sparse Dimension Tuning (SDT), a PEFT method tailored for SSM modules.
  arXiv  Detail & Related papers  (2024-10-11T17:30:28Z)
• SDE: A Simplified and Disentangled Dependency Encoding Framework for State Space Models in Time Series Forecasting [8.841699904757506]
  We identify and formally define three critical dependencies that are fundamental to forecasting accuracy.<n>We propose SDE (Simplified and Disentangled Dependency entangle), a novel framework designed to enhance the capability of SSMs for time series forecasting.
  arXiv  Detail & Related papers  (2024-08-22T02:14:59Z)
• Longhorn: State Space Models are Amortized Online Learners [51.10124201221601]
  State-space models (SSMs) offer linear decoding efficiency while maintaining parallelism during training. In this work, we explore SSM design through the lens of online learning, conceptualizing SSMs as meta-modules for specific online learning problems. We introduce a novel deep SSM architecture, Longhorn, whose update resembles the closed-form solution for solving the online associative recall problem.
  arXiv  Detail & Related papers  (2024-07-19T11:12:08Z)
• Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting [22.84798547604491]
  State Space Models (SSMs) approximate continuous systems using a set of basis functions and discretize them to handle input data. This paper proposes a novel theoretical framework termed Dynamic Spectral Operator, offering more intuitive and general guidance on applying SSMs to time series data. We introduce Time-SSM, a novel SSM-based foundation model with only one-seventh of the parameters compared to Mamba.
  arXiv  Detail & Related papers  (2024-05-25T17:42:40Z)
• HOPE for a Robust Parameterization of Long-memory State Space Models [51.66430224089725]
  State-space models (SSMs) that utilize linear, time-invariant (LTI) systems are known for their effectiveness in learning long sequences. We develop a new parameterization scheme, called HOPE, for LTI systems that utilize Markov parameters within Hankel operators. Our new parameterization endows the SSM with non-decaying memory within a fixed time window, which is empirically corroborated by a sequential CIFAR-10 task with padded noise.
  arXiv  Detail & Related papers  (2024-05-22T20:20:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.