Probing Length Generalization in Mamba via Image Reconstruction

• URL: http://arxiv.org/abs/2603.12499v1
• Date: Thu, 12 Mar 2026 22:32:27 GMT
• Title: Probing Length Generalization in Mamba via Image Reconstruction
• Abstract summary: We show that Mamba's performance can degrade when inference sequence lengths exceed those seen during training.<n>By analyzing reconstructions at different stages of sequence processing, we reveal that Mamba qualitatively adapts its behavior to the distribution of sequence lengths encountered during training.<n>We introduce a length-adaptive variant of Mamba that improves performance across training sequence lengths.
• Score: 0.434964016971127
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mamba has attracted widespread interest as a general-purpose sequence model due to its low computational complexity and competitive performance relative to transformers. However, its performance can degrade when inference sequence lengths exceed those seen during training. We study this phenomenon using a controlled vision task in which Mamba reconstructs images from sequences of image patches. By analyzing reconstructions at different stages of sequence processing, we reveal that Mamba qualitatively adapts its behavior to the distribution of sequence lengths encountered during training, resulting in strategies that fail to generalize beyond this range. To support our analysis, we introduce a length-adaptive variant of Mamba that improves performance across training sequence lengths. Our results provide an intuitive perspective on length generalization in Mamba and suggest directions for improving the architecture.

Related papers

• Rotation Equivariant Mamba for Vision Tasks [66.32081000860958]
  We introduce EQ-VMamba, the first rotation equivariant visual Mamba architecture for vision tasks.<n>We show that EQ-VMamba achieves superior or competitive performance compared to non-equivariant baselines.
  arXiv  Detail & Related papers  (2026-03-10T03:22:33Z)
• Separators in Enhancing Autoregressive Pretraining for Vision Mamba [14.94233154248831]
  We introduce an innovative autoregressive pretraining method for Vision Mamba.<n>New textbfSeparatextbfTors for textbfAutotextbfRegressive pretraining to demarcate and differentiate between different images.
  arXiv  Detail & Related papers  (2026-03-04T07:39:42Z)
• Achilles' Heel of Mamba: Essential difficulties of the Mamba architecture demonstrated by synthetic data [52.07689534063587]
  State Space Models (SSMs) have emerged as promising alternatives to attention mechanisms.<n>In this work, we use carefully designed synthetic tasks to reveal Mamba's inherent limitations.
  arXiv  Detail & Related papers  (2025-09-22T08:38:55Z)
• Detail Matters: Mamba-Inspired Joint Unfolding Network for Snapshot Spectral Compressive Imaging [40.80197280147993]
  We propose a Mamba-inspired Joint Unfolding Network (MiJUN) to overcome the inherent nonlinear and ill-posed characteristics of HSI reconstruction.<n>We introduce an accelerated unfolding network scheme, which reduces the reliance on initial optimization stages.<n>We refine the scanning strategy with Mamba by integrating the tensor mode-$k$ unfolding into the Mamba network.
  arXiv  Detail & Related papers  (2025-01-02T13:56:23Z)
• ReMamba: Equip Mamba with Effective Long-Sequence Modeling [50.530839868893786]
  We propose ReMamba, which enhances Mamba's ability to comprehend long contexts.<n>ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process.
  arXiv  Detail & Related papers  (2024-08-28T02:47:27Z)
• PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training [13.926804198202582]
  Mamba, emerging as a groundbreaking architecture in the field of generative AI, demonstrates remarkable proficiency in handling elongated sequences. Existing training framework of Mamba presents inefficiency with variable-length sequence inputs. We propose PackMamba, a high- throughput Mamba that efficiently handles variable-length sequences.
  arXiv  Detail & Related papers  (2024-08-07T16:13:43Z)
• MambaVision: A Hybrid Mamba-Transformer Vision Backbone [54.965143338206644]
  We propose a novel hybrid Mamba-Transformer backbone, MambaVision, specifically tailored for vision applications.<n>We show that equipping the Mamba architecture with self-attention blocks in the final layers greatly improves its capacity to capture long-range spatial dependencies.<n>For classification on the ImageNet-1K dataset, MambaVision variants achieve state-of-the-art (SOTA) performance in terms of both Top-1 accuracy and throughput.
  arXiv  Detail & Related papers  (2024-07-10T23:02:45Z)
• DeciMamba: Exploring the Length Extrapolation Potential of Mamba [89.07242846058023]
  We introduce DeciMamba, a context-extension method specifically designed for Mamba.<n>Experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths significantly longer than the ones seen during training.
  arXiv  Detail & Related papers  (2024-06-20T17:40:18Z)
• Demystify Mamba in Vision: A Linear Attention Perspective [72.93213667713493]
  Mamba is an effective state space model with linear computation complexity.<n>We show that Mamba shares surprising similarities with linear attention Transformer.<n>We propose a Mamba-Inspired Linear Attention (MILA) model by incorporating the merits of these two key designs into linear attention.
  arXiv  Detail & Related papers  (2024-05-26T15:31:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.