Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

• URL: http://arxiv.org/abs/2606.09080v1
• Date: Mon, 08 Jun 2026 06:26:18 GMT
• Title: Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy
• Abstract summary: Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference.<n>We introduce a GEMM-centric taxonomy that reorganizes existing pruning methods according to the logical textbfM, textbfN, and textbfK dimensions of general matrix multiplication.
• Score: 13.790557800221057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove computation across tokens, layers, heads, dimensions, and attention patterns. Despite sharing the same objective, these pruning approaches induce fundamentally different execution behaviors, causing realized speedups to depend heavily on hardware and kernel implementations. Consequently, the practical acceleration benefits of different pruning families remain poorly understood. In this work, we introduce a GEMM-centric taxonomy that reorganizes existing pruning methods according to the logical \textbf{M}, \textbf{N}, and \textbf{K} dimensions of general matrix multiplication (GEMM). Leveraging this abstraction, we build a unified benchmarking framework that enables implementation-consistent comparison across the pruning design space and systematically characterizes the acceleration--quality Pareto frontier. Our results show that static depth pruning remains the strongest Pareto-optimal baseline and stays closest to its theoretical acceleration upper bound in memory-bounded scenarios. During prefill, the frontier transitions from static depth at low quality loss (0\%--4\%), to dynamic depth at moderate loss (5\%--16\%), and finally to static width pruning at higher loss levels (17\%--26\%). These findings establish the first unified view of the practical limits of pruning-based LLM acceleration and provide guidance for future pruning research.\footnote{Code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/PruningInferSim}

Related papers

• CRISP: Cliff-awaRe Input-adaptive Sparse Prefilling with Structural-Mass-Motivated Routing [90.63350360476294]
  We present CRISP (Cliff-awaRe Input-adaptive Sparse Prefilling), which identifies and addresses two structural challenges in this dynamic routing paradigm.<n>We show that the routing decision can be read directly off the structure of the proxy attention map.<n>We formalize the post-softmax mass cliff and demonstrate theoretically that strictly cumulative coverage thresholds accumulate O(n) background noise at long contexts.
  arXiv  Detail & Related papers  (2026-09-01T22:49:46Z)
• Thought-Level Beam Search for Reasoning [39.76902449987798]
  Test-time compute scaling is a primary driver of performance in large reasoning models.<n>We formalize test-time reasoning as a constrained compute allocation problem over partial trajectories.<n>We introduce Gambit, an inference algorithm that executes emphthought-level beam search
  arXiv  Detail & Related papers  (2026-08-08T09:00:01Z)
• Alternating Gradient Flow Utility: A Unified Metric for Structural Pruning and Dynamic Routing in Deep Networks [52.153950303594684]
  We propose a decoupled kinetic paradigm inspired by Alternating Gradient Flow (AGF)<n>AGF accurately captures the network's structural "kinetic utility"<n>We design a hybrid routing framework that decouples AGF-guided offline structural search from online execution via zero-cost physical priors.
  arXiv  Detail & Related papers  (2026-03-12T18:19:21Z)
• FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach [58.345210583013454]
  FineRMoE is an architecture that extends fine-grained expert design to both intermediate and output dimensions.<n>It achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
  arXiv  Detail & Related papers  (2026-03-09T12:00:36Z)
• MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios [12.409831039709886]
  Mixture-of-Experts (MoE) models enable scalable performance but face severe memory constraints on edge devices.<n>We introduce MoE-SpAc, an MoE inference framework that integrates a Speculative Utility Estor to track expert demand.<n>Experiments on seven benchmarks demonstrate that MoE-SpAc achieves a 42% improvement in TPS over the SOTA SD-based baseline.
  arXiv  Detail & Related papers  (2026-02-12T04:28:00Z)
• Revisiting Weighted Strategy for Non-stationary Parametric Bandits and MDPs [56.246783503873225]
  This paper revisits the weighted strategy for non-stationary parametric bandits.<n>We propose a simpler weight-based algorithm that is as efficient as window/restart-based algorithms.<n>Our framework can be used to improve regret bounds of other parametric bandits.
  arXiv  Detail & Related papers  (2026-01-03T04:50:21Z)
• StructPrune: Structured Global Pruning asymptotics with $\mathcal{O}(\sqrt{N})$ GPU Memory [11.996799691784693]
  Pruning is critical for scaling large language models (LLMs)<n>Global pruning achieves strong performance but requires $mathcalO(N)$ memory.<n>Local pruning reduces GPU memory usage to that of a single layer by pruning layers independently.
  arXiv  Detail & Related papers  (2025-09-25T19:16:50Z)
• Effective Dimension Aware Fractional-Order Stochastic Gradient Descent for Convex Optimization Problems [2.5971517743176915]
  We propose 2SED Fractional-Order Gradient Descent (2SEDFOSGD) to adapt the fractional exponent in a data-driven manner.<n>Theoretically, this approach preserves the advantages of fractional memory without the sluggish or unstable behavior observed in na"ive fractional SGD.
  arXiv  Detail & Related papers  (2025-03-17T22:57:37Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescaling [6.8271468231014145]
  We investigate margin-maximization bias by gradient-based algorithms in classifying linearly separable data.<n>We propose a novel algorithm called Progressive Rescaling Gradient (PRGD) and show that PRGD can maximize the margin at an em exponential rate<n>PRGD also shows promise in enhancing the generalization performance when applied to linearly non-separable datasets and deep neural networks.
  arXiv  Detail & Related papers  (2023-11-24T10:07:10Z)
• A Unified Framework for Soft Threshold Pruning [27.853698217792456]
  We reformulate soft threshold pruning as an implicit optimization problem solved using the Iterative Shrinkage-Thresholding Algorithm (ISTA) We derive an optimal threshold scheduler through an in-depth study of threshold scheduling based on our framework. In principle, the derived pruning algorithm could sparsify any mathematical model trained via SGD.
  arXiv  Detail & Related papers  (2023-02-25T08:16:14Z)
• Dynamic Probabilistic Pruning: A general framework for hardware-constrained pruning at different granularities [80.06422693778141]
  We propose a flexible new pruning mechanism that facilitates pruning at different granularities (weights, kernels, filters/feature maps) We refer to this algorithm as Dynamic Probabilistic Pruning (DPP) We show that DPP achieves competitive compression rates and classification accuracy when pruning common deep learning models trained on different benchmark datasets for image classification.
  arXiv  Detail & Related papers  (2021-05-26T17:01:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.