S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devices

• URL: http://arxiv.org/abs/2606.18096v1
• Date: Tue, 16 Jun 2026 15:59:10 GMT
• Title: S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devices
• Abstract summary: We propose a novel incremental, operator-level pruning approach for S4- and S4D-based models.<n>Our method progressively prunes model operators by interleaving structured masking with fine-tuning.<n>Experiments show that pruning up to 70% of the model operators preserves the performance of the original models in most cases.
• Score: 3.231646967692886
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Structured State Space Models (SSMs), including the S4 and S4D architectures, have recently emerged as powerful alternatives to attention-based models for capturing long-range dependencies in sequential data. Despite their strong empirical performance, deploying these models in time- and resource-constrained settings remains challenging due to their computational and memory demands. In this paper, we propose a novel incremental, operator-level pruning approach for S4- and S4D-based models that significantly reduces inference cost while preserving predictive performance. To the best of our knowledge, this is the first work to systematically investigate structured operator pruning for SSMs. Our method progressively prunes model operators by interleaving structured masking with fine-tuning, while jointly monitoring accuracy and inference latency. We implement this approach within a unified training and evaluation framework that enables systematic exploration of efficiency-accuracy trade-offs. Experiments across multiple benchmark datasets show that pruning up to 70% of the model operators preserves the performance of the original models in most cases, while substantially reducing inference latency. These results demonstrate that structured operator pruning is an effective and previously unexplored strategy for improving the efficiency of SSMs and facilitate their deployment in practical, resource-constrained scenarios.

Related papers

• AgentCPM-Explore: Realizing Long-Horizon Deep Exploration for Edge-Scale Agents [75.67445299298949]
  AgentCPM-Explore is a compact 4B agent model with high knowledge density and strong exploration capability.<n>We introduce a holistic training framework featuring parameter-space model fusion, reward signal denoising, and contextual information refinement.<n>AgentCPM-Explore achieves state-of-the-art (SOTA) performance among 4B-class models, matches or surpasses 8B-class SOTA models on four benchmarks, and even outperforms larger-scale models such as Claude-4.5-Sonnet or DeepSeek-v3.2 in five benchmarks.
  arXiv  Detail & Related papers  (2026-02-06T08:24:59Z)
• CALM Before the STORM: Unlocking Native Reasoning for Optimization Modeling [60.55856973678002]
  Large Reasoning Models (LRMs) have demonstrated strong capabilities in complex multi-step reasoning.<n>Existing domain adaptation methods, originally designed for earlier instruction-tuned models, often fail to exploit the advanced reasoning patterns of modern LRMs.<n>We propose textbfCALM, a framework that progressively refines LRMs within their native reasoning modes for optimization modeling tasks.
  arXiv  Detail & Related papers  (2025-10-05T13:38:31Z)
• The Curious Case of In-Training Compression of State Space Models [49.819321766705514]
  State Space Models (SSMs) tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference.<n>Key design challenge is striking the right balance between maximizing expressivity and limiting this computational burden.<n>Our approach, textscCompreSSM, applies to Linear Time-Invariant SSMs such as Linear Recurrent Units, but is also extendable to selective models.
  arXiv  Detail & Related papers  (2025-10-03T09:02:33Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• On the Simplification of Neural Network Architectures for Predictive Process Monitoring [0.5735035463793009]
  We analyze how reducing model complexity, both in terms of parameter count and architectural depth, impacts predictive performance.<n>We show that shrinking the Transformer model by 85% results in only a 2-3% drop in performance.<n>Overall, our findings suggest that substantial model simplification can preserve predictive accuracy.
  arXiv  Detail & Related papers  (2025-09-21T16:21:45Z)
• W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling [23.453158933852357]
  State Space Models (SSMs) have emerged as powerful components for sequence modeling.<n>We introduce a new variant, W4S4 (WaLRUS for S4), a new class of SSMs constructed from redundant wavelet frames.<n>We show that WaLRUS retains information over long horizons significantly better than HiPPO-based SSMs.
  arXiv  Detail & Related papers  (2025-06-09T16:33:29Z)
• From S4 to Mamba: A Comprehensive Survey on Structured State Space Models [0.0]
  Recent advancements in sequence modeling have led to the emergence of Structured State Space Models (SSMs)<n>SSMs leverage structured recurrence and state-space representations to achieve superior long-sequence processing with linear or near-linear complexity.<n>This survey serves as a structured guide for researchers and practitioners, detailing the advancements, trade-offs, and future directions of SSM-based architectures in AI and deep learning.
  arXiv  Detail & Related papers  (2025-03-22T01:55:32Z)
• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.<n>We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively [10.812755570974929]
  We use Model Structural Redundancy Score (MSRS) to measure the degree of redundancy in a deep learning model structure. MSRS is effective in both revealing and assessing the redundancy issues in many state-of-the-art models. We design a novel redundancy-aware algorithm to guide the search for the optimal model structure.
  arXiv  Detail & Related papers  (2024-11-15T14:36:07Z)
• Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning [78.72226641279863]
  Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. Our research explores task-specific model pruning to inform decisions about designing SMoE architectures. We introduce an adaptive task-aware pruning technique UNCURL to reduce the number of experts per MoE layer in an offline manner post-training.
  arXiv  Detail & Related papers  (2024-09-02T22:35:03Z)
• HOPE for a Robust Parameterization of Long-memory State Space Models [51.66430224089725]
  State-space models (SSMs) that utilize linear, time-invariant (LTI) systems are known for their effectiveness in learning long sequences. We develop a new parameterization scheme, called HOPE, for LTI systems that utilize Markov parameters within Hankel operators. Our new parameterization endows the SSM with non-decaying memory within a fixed time window, which is empirically corroborated by a sequential CIFAR-10 task with padded noise.
  arXiv  Detail & Related papers  (2024-05-22T20:20:14Z)
• Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation [0.0]
  We propose to use the balanced truncation, a prevalent model reduction technique in control theory, applied specifically to DSS layers in pre-trained S4 model as a novel model compression method. Numerical experiments demonstrate that our trained models combined with the balanced truncation surpass conventionally trained models with Skew-HiPPO.
  arXiv  Detail & Related papers  (2024-02-25T05:22:45Z)
• EsaCL: Efficient Continual Learning of Sparse Models [10.227171407348326]
  Key challenge in the continual learning setting is to efficiently learn a sequence of tasks without forgetting how to perform previously learned tasks. We propose a new method for efficient continual learning of sparse models (EsaCL) that can automatically prune redundant parameters without adversely impacting the model's predictive power.
  arXiv  Detail & Related papers  (2024-01-11T04:59:44Z)
• Structured State Space Models for In-Context Reinforcement Learning [30.189834820419446]
  Structured state space sequence (S4) models have recently achieved state-of-the-art performance on long-range sequence modeling tasks. We propose a modification to a variant of S4 that enables us to initialise and reset the hidden state in parallel. We show that our modified architecture runs faster than Transformers in sequence length and performs better than RNN's on a simple memory-based task.
  arXiv  Detail & Related papers  (2023-03-07T15:32:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.