DAMamba: Vision State Space Model with Dynamic Adaptive Scan

• URL: http://arxiv.org/abs/2502.12627v1
• Date: Tue, 18 Feb 2025 08:12:47 GMT
• Title: DAMamba: Vision State Space Model with Dynamic Adaptive Scan
• Authors: Tanzhe Li, Caoshuo Li, Jiayi Lyu, Hongjuan Pei, Baochang Zhang, Taisong Jin, Rongrong Ji,
• Abstract summary: State space models (SSMs) have recently garnered significant attention in computer vision. We propose Dynamic Adaptive Scan (DAS), a data-driven method that adaptively allocates scanning orders and regions. Based on DAS, we propose the vision backbone DAMamba, which significantly outperforms current state-of-the-art vision Mamba models in vision tasks.
• Score: 51.81060691414399
• License:
• Abstract: State space models (SSMs) have recently garnered significant attention in computer vision. However, due to the unique characteristics of image data, adapting SSMs from natural language processing to computer vision has not outperformed the state-of-the-art convolutional neural networks (CNNs) and Vision Transformers (ViTs). Existing vision SSMs primarily leverage manually designed scans to flatten image patches into sequences locally or globally. This approach disrupts the original semantic spatial adjacency of the image and lacks flexibility, making it difficult to capture complex image structures. To address this limitation, we propose Dynamic Adaptive Scan (DAS), a data-driven method that adaptively allocates scanning orders and regions. This enables more flexible modeling capabilities while maintaining linear computational complexity and global modeling capacity. Based on DAS, we further propose the vision backbone DAMamba, which significantly outperforms current state-of-the-art vision Mamba models in vision tasks such as image classification, object detection, instance segmentation, and semantic segmentation. Notably, it surpasses some of the latest state-of-the-art CNNs and ViTs. Code will be available at https://github.com/ltzovo/DAMamba.

Related papers

• Mamba2D: A Natively Multi-Dimensional State-Space Model for Vision Tasks [47.49096400786856]
  State-Space Models (SSMs) have recently emerged as a powerful and efficient alternative to the long-standing transformer architecture. We re-deriving modern selective state-space techniques, starting from a multidimensional formulation. Mamba2D shows comparable performance to prior adaptations of SSMs for vision tasks, on standard image classification evaluations with the ImageNet-1K dataset.
  arXiv  Detail & Related papers  (2024-12-20T18:50:36Z)
• Efficient Visual State Space Model for Image Deblurring [83.57239834238035]
  Convolutional neural networks (CNNs) and Vision Transformers (ViTs) have achieved excellent performance in image restoration. We propose a simple yet effective visual state space model (EVSSM) for image deblurring.
  arXiv  Detail & Related papers  (2024-05-23T09:13:36Z)
• RSMamba: Remote Sensing Image Classification with State Space Model [25.32283897448209]
  We introduce RSMamba, a novel architecture for remote sensing image classification. RSMamba is based on the State Space Model (SSM) and incorporates an efficient, hardware-aware design known as the Mamba. We propose a dynamic multi-path activation mechanism to augment Mamba's capacity to model non-temporal image data.
  arXiv  Detail & Related papers  (2024-03-28T17:59:49Z)
• VmambaIR: Visual State Space Model for Image Restoration [36.11385876754612]
  We propose VmambaIR, which introduces State Space Models (SSMs) with linear complexity into comprehensive image restoration tasks. VmambaIR achieves state-of-the-art (SOTA) performance with much fewer computational resources and parameters.
  arXiv  Detail & Related papers  (2024-03-18T02:38:55Z)
• LocalMamba: Visual State Space Model with Windowed Selective Scan [45.00004931200446]
  Key to enhancing Vision Mamba (ViM) lies in optimizing scan directions for sequence modeling. We introduce a novel local scanning strategy that divides images into distinct windows, effectively capturing local dependencies. Our model significantly outperforms Vim-Ti by 3.1% on ImageNet with the same 1.5G FLOPs.
  arXiv  Detail & Related papers  (2024-03-14T12:32:40Z)
• nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Model [24.955052600683423]
  In this paper, we introduce nnMamba, a novel architecture that integrates the strengths of CNNs and the advanced long-range modeling capabilities of State Space Sequence Models (SSMs) Experiments on 6 datasets demonstrate nnMamba's superiority over state-of-the-art methods in a suite of challenging tasks, including 3D image segmentation, classification, and landmark detection.
  arXiv  Detail & Related papers  (2024-02-05T21:28:47Z)
• VMamba: Visual State Space Model [98.0517369083152]
  We adapt Mamba, a state-space language model, into VMamba, a vision backbone with linear time complexity. At the core of VMamba is a stack of Visual State-Space (VSS) blocks with the 2D Selective Scan (SS2D) module.
  arXiv  Detail & Related papers  (2024-01-18T17:55:39Z)
• Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model [48.233300343211205]
  We propose a new generic vision backbone with bidirectional Mamba blocks (Vim) Vim marks the image sequences with position embeddings and compresses the visual representation with bidirectional state space models. The results demonstrate that Vim is capable of overcoming the computation & memory constraints on performing Transformer-style understanding for high-resolution images.
  arXiv  Detail & Related papers  (2024-01-17T18:56:18Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.