Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing

• URL: http://arxiv.org/abs/2502.00594v1
• Date: Sat, 01 Feb 2025 23:35:20 GMT
• Title: Fast Vision Mamba: Pooling Spatial Dimensions for Accelerated Processing
• Authors: Saarthak Kapse, Robin Betz, Srinivasan Sivanandan,
• Abstract summary: State Space Models (SSMs) with selective scan (Mamba) have been adapted into efficient vision models.<n>Fast Vision Mamba (FastVim) reduces the number of recurrent steps in Vision Mamba models while still retaining model performance.<n>Our experiments demonstrate state-of-the-art performance with dramatically improved throughput in a range of tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: State Space Models (SSMs) with selective scan (Mamba) have been adapted into efficient vision models. Mamba, unlike Vision Transformers, achieves linear complexity for token interactions through a recurrent hidden state process. This sequential processing is enhanced by a parallel scan algorithm, which reduces the computational time of recurrent steps from $L$ sequential steps to $log(L)$ parallel steps with respect to the number of input tokens ($L$). In this work, we propose Fast Vision Mamba (FastVim), that further reduces the computational time of the SSM block by reducing the number of recurrent steps in Vision Mamba models while still retaining model performance. By alternately pooling tokens along image dimensions across Mamba blocks, we obtain a 2$\times$ reduction in the number of parallel steps in SSM block. Our model offers up to $72.5\%$ speedup in inference speed compared to baseline Vision Mamba models on high resolution (2048$\times$2048) images. Our experiments demonstrate state-of-the-art performance with dramatically improved throughput in a range of tasks such as image classification, cell perturbation prediction, segmentation, and object detection. Code is made available at https://github.com/insitro/FastVim

Related papers

• DefMamba: Deformable Visual State Space Model [65.50381013020248]
  We propose a novel visual foundation model called DefMamba. By combining a deformable scanning(DS) strategy, this model significantly improves its ability to learn image structures and detects changes in object details. Numerous experiments have shown that DefMamba achieves state-of-the-art performance in various visual tasks.
  arXiv  Detail & Related papers  (2025-04-08T08:22:54Z)
• Dynamic Vision Mamba [41.84910346271891]
  Mamba-based vision models have gained extensive attention as a result of being computationally more efficient than attention-based models. For token redundancy, we analytically find that early token pruning methods will result in inconsistency between training and inference. For block redundancy, we allow each image to select SSM blocks dynamically based on an empirical observation that the inference speed of Mamba-based vision models is largely affected by the number of SSM blocks.
  arXiv  Detail & Related papers  (2025-04-07T07:31:28Z)
• MatIR: A Hybrid Mamba-Transformer Image Restoration Model [95.17418386046054]
  We propose a Mamba-Transformer hybrid image restoration model called MatIR. MatIR cross-cycles the blocks of the Transformer layer and the Mamba layer to extract features. In the Mamba module, we introduce the Image Inpainting State Space (IRSS) module, which traverses along four scan paths.
  arXiv  Detail & Related papers  (2025-01-30T14:55:40Z)
• 2DMamba: Efficient State Space Model for Image Representation with Applications on Giga-Pixel Whole Slide Image Classification [40.10133518650528]
  Mamba introduced a selective State Space Model (SSM) with linear complexity and high parallelism.<n>We propose 2DMamba, a novel 2D selective SSM framework that incorporates the 2D spatial structure of images into Mamba.<n>Experiments on 10 public datasets for WSI classification and survival analysis show that 2DMambaimproves up to $2.48%$ in AUC, $3.11%$ in F1 score, $2.47%$ in accuracy and $5.52%$ in C-index.
  arXiv  Detail & Related papers  (2024-12-01T05:42:58Z)
• V2M: Visual 2-Dimensional Mamba for Image Representation Learning [68.51380287151927]
  Mamba has garnered widespread attention due to its flexible design and efficient hardware performance to process 1D sequences. Recent studies have attempted to apply Mamba to the visual domain by flattening 2D images into patches and then regarding them as a 1D sequence. We propose a Visual 2-Dimensional Mamba model as a complete solution, which directly processes image tokens in the 2D space.
  arXiv  Detail & Related papers  (2024-10-14T11:11:06Z)
• Scalable Autoregressive Image Generation with Mamba [23.027439743155192]
  We introduce AiM, an autoregressive (AR) image generative model based on Mamba architecture.<n>Mamba is a novel state-space model characterized by its exceptional performance for long-sequence modeling with linear time.<n>We provide AiM models in various scales, with parameter counts ranging from 148M to 1.3B.
  arXiv  Detail & Related papers  (2024-08-22T09:27:49Z)
• PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training [13.926804198202582]
  Mamba, emerging as a groundbreaking architecture in the field of generative AI, demonstrates remarkable proficiency in handling elongated sequences. Existing training framework of Mamba presents inefficiency with variable-length sequence inputs. We propose PackMamba, a high- throughput Mamba that efficiently handles variable-length sequences.
  arXiv  Detail & Related papers  (2024-08-07T16:13:43Z)
• DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis [56.849285913695184]
  Diffusion Mamba (DiM) is a sequence model for efficient high-resolution image synthesis. DiM architecture achieves inference-time efficiency for high-resolution images. Experiments demonstrate the effectiveness and efficiency of our DiM.
  arXiv  Detail & Related papers  (2024-05-23T06:53:18Z)
• PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition [21.761988930589727]
  PlainMamba is a simple non-hierarchical state space model (SSM) designed for general visual recognition. We adapt the selective scanning process of Mamba to the visual domain, enhancing its ability to learn features from two-dimensional images. Our architecture is designed to be easy to use and easy to scale, formed by stacking identical PlainMamba blocks.
  arXiv  Detail & Related papers  (2024-03-26T13:35:10Z)
• MambaByte: Token-free Selective State Space Model [71.90159903595514]
  MambaByte is a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. We show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks.
  arXiv  Detail & Related papers  (2024-01-24T18:53:53Z)
• VMamba: Visual State Space Model [98.0517369083152]
  We adapt Mamba, a state-space language model, into VMamba, a vision backbone with linear time complexity.<n>At the core of VMamba is a stack of Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module.
  arXiv  Detail & Related papers  (2024-01-18T17:55:39Z)
• Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model [48.233300343211205]
  We propose a new generic vision backbone with bidirectional Mamba blocks (Vim) Vim marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images.
  arXiv  Detail & Related papers  (2024-01-17T18:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.