Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments

• URL: http://arxiv.org/abs/2505.08299v1
• Date: Tue, 13 May 2025 07:23:08 GMT
• Title: Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments
• Authors: Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma,
• Abstract summary: State-space models (SSMs) have emerged as powerful alternatives to Transformers for sequence modeling.<n>We propose a novel unstructured pruning framework tailored for Mamba models that achieves up to 70% parameter reduction while retaining over 95% of the original performance.
• Score: 2.1797343876622097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State-space models (SSMs), particularly the Mamba architecture, have emerged as powerful alternatives to Transformers for sequence modeling, offering linear-time complexity and competitive performance across diverse tasks. However, their large parameter counts pose significant challenges for deployment in resource-constrained environments. We propose a novel unstructured pruning framework tailored for Mamba models that achieves up to 70\% parameter reduction while retaining over 95\% of the original performance. Our approach integrates three key innovations: (1) a gradient-aware magnitude pruning technique that combines weight magnitude and gradient information to identify less critical parameters, (2) an iterative pruning schedule that gradually increases sparsity to maintain model stability, and (3) a global pruning strategy that optimizes parameter allocation across the entire model. Through extensive experiments on WikiText-103, Long Range Arena, and ETT time-series benchmarks, we demonstrate significant efficiency gains with minimal performance degradation. Our analysis of pruning effects on Mamba's components reveals critical insights into the architecture's redundancy and robustness, enabling practical deployment in resource-constrained settings while broadening Mamba's applicability.

Related papers

• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection [88.47928738482719]
  Linear State Space Models (SSMs) offer remarkable performance gains in sequence modeling.<n>Recent advances, such as Mamba, further enhance SSMs with input-dependent gating and hardware-aware implementations.<n>We introduce Routing Mamba (RoM), a novel approach that scales SSM parameters using sparse mixtures of linear projection experts.
  arXiv  Detail & Related papers  (2025-06-22T19:26:55Z)
• Scaling Reasoning without Attention [44.42046576158219]
  We introduce ourmodel, an attention-free language model that addresses both issues through architectural and data-centric innovations.<n>Our model eliminates the need for self-attention and key-value caching, enabling fixed-memory, constant-time inference.<n>On benchmark evaluations, ourmodel-7B outperforms strong Transformer and hybrid models of comparable scale.
  arXiv  Detail & Related papers  (2025-05-28T14:52:15Z)
• RD-UIE: Relation-Driven State Space Modeling for Underwater Image Enhancement [59.364418120895]
  Underwater image enhancement (UIE) is a critical preprocessing step for marine vision applications.<n>We develop a novel relation-driven Mamba framework for effective UIE (RD-UIE)<n>Experiments on underwater enhancement benchmarks demonstrate RD-UIE outperforms the state-of-the-art approach WMamba.
  arXiv  Detail & Related papers  (2025-05-02T12:21:44Z)
• Efficient Multi-Instance Generation with Janus-Pro-Dirven Prompt Parsing [53.295515505026096]
  Janus-Pro-driven Prompt Parsing is a prompt- parsing module that bridges text understanding and layout generation.<n>MIGLoRA is a parameter-efficient plug-in integrating Low-Rank Adaptation into UNet (SD1.5) and DiT (SD3) backbones.<n>The proposed method achieves state-of-the-art performance on COCO and LVIS benchmarks while maintaining parameter efficiency.
  arXiv  Detail & Related papers  (2025-03-27T00:59:14Z)
• RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing [28.488986896516284]
  RoMA is a framework that enables scalable self-supervised pretraining of RS foundation models using large-scale, diverse, unlabeled data.<n>RoMA enhances scalability for high-resolution images through a tailored auto-regressive learning strategy.<n> experiments across scene classification, object detection, and semantic segmentation tasks demonstrate that RoMA-pretrained Mamba models consistently outperform ViT-based counterparts in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-03-13T14:09:18Z)
• SegResMamba: An Efficient Architecture for 3D Medical Image Segmentation [2.979183050755201]
  We propose an efficient 3D segmentation model for medical imaging called SegResMamba.<n>Our model uses less than half the memory during training compared to other state-of-the-art (SOTA) architectures.
  arXiv  Detail & Related papers  (2025-03-10T18:40:28Z)
• Rethinking Token Reduction for State Space Models [47.00760373683448]
  We propose a tailored, unified post-training token reduction method for State Space Models (SSMs) Our approach integrates token importance and similarity, thus taking advantage of both pruning and merging. Our method improves the average accuracy by 5.7% to 13.1% on six benchmarks with Mamba-2 compared to existing methods.
  arXiv  Detail & Related papers  (2024-10-16T00:06:13Z)
• Mamba or Transformer for Time Series Forecasting? Mixture of Universals (MoU) Is All You Need [28.301119776877822]
  Time series forecasting requires balancing short-term and long-term dependencies for accurate predictions. Transformers are superior in modeling long-term dependencies but are criticized for their quadratic computational cost. Mamba provides a near-linear alternative but is reported less effective in time series longterm forecasting due to potential information loss.
  arXiv  Detail & Related papers  (2024-08-28T17:59:27Z)
• ReMamba: Equip Mamba with Effective Long-Sequence Modeling [50.530839868893786]
  We propose ReMamba, which enhances Mamba's ability to comprehend long contexts.<n>ReMamba incorporates selective compression and adaptation techniques within a two-stage re-forward process.
  arXiv  Detail & Related papers  (2024-08-28T02:47:27Z)
• SIGMA: Selective Gated Mamba for Sequential Recommendation [56.85338055215429]
  Mamba, a recent advancement, has exhibited exceptional performance in time series prediction.<n>We introduce a new framework named Selective Gated Mamba ( SIGMA) for Sequential Recommendation.<n>Our results indicate that SIGMA outperforms current models on five real-world datasets.
  arXiv  Detail & Related papers  (2024-08-21T09:12:59Z)
• Can pruning make Large Language Models more efficient? [0.0]
  This paper investigates the application of weight pruning as an optimization strategy for Transformer architectures. Our findings suggest that significant reductions in model size are attainable without considerable compromise on performance. This work seeks to bridge the gap between model efficiency and performance, paving the way for more scalable and environmentally responsible deep learning applications.
  arXiv  Detail & Related papers  (2023-10-06T20:28:32Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.