Swimba: Switch Mamba Model Scales State Space Models

• URL: http://arxiv.org/abs/2603.06938v1
• Date: Fri, 06 Mar 2026 23:15:04 GMT
• Title: Swimba: Switch Mamba Model Scales State Space Models
• Abstract summary: Mixture-of-experts (MoE) is a common approach for increasing parameter capacity.<n>Applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent state update.<n>We show how to introduce expert specialization into selective SSMs while preserving computational efficiency.
• Score: 10.385416238048503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-experts (MoE) is a common approach for increasing parameter capacity, but applying MoE to state space model (SSM) token mixers can multiply the cost of the recurrent state update. We study how to introduce expert specialization into selective SSMs while preserving computational efficiency. We show that MoE--SSM can refer to two designs: (1) MoE over separated SSMs, which maintains multiple state trajectories and thus scales compute with the number of experts; and (2) MoE-parameterized SSM, which mixes experts in parameter space, maintains a single state trajectory, and evaluates the recurrence once. Our method, Switch Mamba (Swimba), follows the second design by routing over expert-produced SSM streams. Theoretically, we establish well-definedness and stability for MoE-parameterized SSMs and characterize the relationship between the two designs. Empirically, we evaluate Swimba on standard benchmark tasks and measure real-time throughput and latency. Under matched FLOPs, Swimba achieves slightly better average performance than the baseline, with a small slowdown in real-time latency and throughput. Overall, these results suggest that parameter-space MoE can increase SSM capacity while keeping the dominant recurrence cost fixed.

Related papers

• Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification [50.994194925685434]
  We show that a looped SSM with $k$ parameters iterated $L$ times consistently closely matches or outperforms a standard SSM.<n>We also show that input reshaping is an equally neglected design axis.
  arXiv  Detail & Related papers  (2026-05-15T15:18:12Z)
• MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling [60.648359990090846]
  State-space models (SSMs) have recently attention as an efficient alternative to computationally expensive attention-based models for sequence modeling.<n>This paper introduces a multi-scale SSM framework that represents sequence dynamics across multiple resolution and processing each resolution with specialized state-space dynamics.
  arXiv  Detail & Related papers  (2025-12-29T19:36:28Z)
• First-order State Space Model for Lightweight Image Super-resolution [31.445028679758106]
  State space models (SSMs) have shown promise in NLP tasks and are increasingly applied to vision tasks.<n>In this paper, we introduce the First-order State Space Model (FSSM) to improve the original Mamba module.<n>Extensive experimental results demonstrate that FSSM improves the performance of MambaIR on five benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-10T10:00:43Z)
• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection [88.47928738482719]
  Linear State Space Models (SSMs) offer remarkable performance gains in sequence modeling.<n>Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations.<n>We introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts.
  arXiv  Detail & Related papers  (2025-06-22T19:26:55Z)
• QMamba: Post-Training Quantization for Vision State Space Models [45.97843526485619]
  State Space Models (SSMs) have gained increasing attention for vision models recently.<n>Given the computational cost of deploying SSMs on resource-limited edge devices, Post-Training Quantization (PTQ) is a technique with the potential for efficient deployment of SSMs.<n>We propose QMamba, one of the first PTQ frameworks to be designed for vision SSMs based on the analysis of the activation distributions in SSMs.
  arXiv  Detail & Related papers  (2025-01-23T12:45:20Z)
• Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning [54.19222454702032]
  Continual Learning aims to equip AI models with the ability to learn a sequence of tasks over time, without forgetting previously learned knowledge.<n>State Space Models (SSMs) have achieved notable success in computer vision.<n>We introduce Mamba-CL, a framework that continuously fine-tunes the core SSMs of the large-scale Mamba foundation model.
  arXiv  Detail & Related papers  (2024-11-23T06:36:16Z)
• Provable Benefits of Complex Parameterizations for Structured State Space Models [51.90574950170374]
  Structured state space models (SSMs) are linear dynamical systems adhering to a specified structure. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. This paper takes a step towards explaining the benefits of complex parameterizations for SSMs by establishing formal gaps between real and complex diagonal SSMs.
  arXiv  Detail & Related papers  (2024-10-17T22:35:50Z)
• KalMamba: Towards Efficient Probabilistic State Space Models for RL under Uncertainty [18.611360495409087]
  Probabilistic State Space Models (SSMs) are essential for Reinforcement Learning (RL) from high-dimensional, partial information as they provide concise representations for control. We propose KalMamba, an efficient architecture to learn representations for RL that combines the strengths of probabilistic SSMs with the scalability of deterministic SSMs.
  arXiv  Detail & Related papers  (2024-06-21T13:27:36Z)
• SMR: State Memory Replay for Long Sequence Modeling [19.755738298836526]
  This paper proposes a novel non-recursive non-uniform sample processing strategy to overcome compatibility limitations in parallel convolutional computation. We introduce State Memory Replay (SMR), which utilizes learnable memories to adjust the current state with multi-step information for generalization at sampling points different from those in the training data. Experiments on long-range modeling tasks in autoregressive language modeling and Long Range Arena demonstrate the general effectiveness of the SMR mechanism for a series of SSM models.
  arXiv  Detail & Related papers  (2024-05-27T17:53:32Z)
• HOPE for a Robust Parameterization of Long-memory State Space Models [51.66430224089725]
  State-space models (SSMs) that utilize linear, time-invariant (LTI) systems are known for their effectiveness in learning long sequences. We develop a new parameterization scheme, called HOPE, for LTI systems that utilize Markov parameters within Hankel operators. Our new parameterization endows the SSM with non-decaying memory within a fixed time window, which is empirically corroborated by a sequential CIFAR-10 task with padded noise.
  arXiv  Detail & Related papers  (2024-05-22T20:20:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.