Sparse Computations in Deep Learning Inference

• URL: http://arxiv.org/abs/2512.02550v1
• Date: Tue, 02 Dec 2025 09:19:33 GMT
• Title: Sparse Computations in Deep Learning Inference
• Authors: Ioanna Tasou, Panagiotis Mpakos, Angelos Vlachos, Dionysios Adamopoulos, Georgios Giannakopoulos, Konstantinos Katsikopoulos, Ioannis Karaparisis, Maria Lazou, Spyridon Loukovitis, Areti Mei, Anastasia Poulopoulou, Angeliki Dimitriou, Giorgos Filandrianos, Dimitrios Galanopoulos, Vasileios Karampinis, Ilias Mitsouras, Nikolaos Spanos, Petros Anastasiadis, Ioannis Doudalis, Konstantinos Nikas, George Retsinas, Paraskevi Tzouveli, Christina Giannoula, Nectarios Koziris, Nikela Papadopoulou, Giorgos Stamou, Athanasios Voulodimos, Georgios Goumas,
• Abstract summary: Sparsity stands out as a critical mechanism for drastically reducing inference demands.<n>It is largely untapped and is not yet fully incorporated in production AI systems.<n>This paper aims to serve as a resource for performance engineers seeking to develop and deploy highly efficient sparse deep learning models.
• Score: 14.210576143844435
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The computational demands of modern Deep Neural Networks (DNNs) are immense and constantly growing. While training costs usually capture public attention, inference demands are also contributing in significant computational, energy and environmental footprints. Sparsity stands out as a critical mechanism for drastically reducing these resource demands. However, its potential remains largely untapped and is not yet fully incorporated in production AI systems. To bridge this gap, this work provides the necessary knowledge and insights for performance engineers keen to get involved in deep learning inference optimization. In particular, in this work we: a) discuss the various forms of sparsity that can be utilized in DNN inference, b) explain how the original dense computations translate to sparse kernels, c) provide an extensive bibliographic review of the state-of-the-art in the implementation of these kernels for CPUs and GPUs, d) discuss the availability of sparse datasets in support of sparsity-related research and development, e) explore the current software tools and frameworks that provide robust sparsity support, and f) present evaluation results of different implementations of the key SpMM and SDDMM kernels on CPU and GPU platforms. Ultimately, this paper aims to serve as a resource for performance engineers seeking to develop and deploy highly efficient sparse deep learning models in productions.

Related papers

• Towards Automated Kernel Generation in the Era of LLMs [17.69471168609145]
  Kernel engineering is a time-consuming and non-scalable process.<n>Recent advances in large language models (LLMs) and agentic systems have opened new possibilities for automating kernel generation and optimization.<n>The field remains fragmented, lacking a systematic perspective for LLM-driven kernel generation.
  arXiv  Detail & Related papers  (2026-01-22T07:53:52Z)
• GOGH: Correlation-Guided Orchestration of GPUs in Heterogeneous Clusters [4.241410532880399]
  We propose a learning-based architecture for managing machine learning workloads in heterogeneous clusters.<n>The system operates online, allocating resources to incoming training or inference requests while minimizing energy consumption.
  arXiv  Detail & Related papers  (2025-10-17T13:44:10Z)
• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals [58.83169560132308]
  We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks.
  arXiv  Detail & Related papers  (2024-07-18T17:59:01Z)
• Enhancing Dropout-based Bayesian Neural Networks with Multi-Exit on FPGA [20.629635991749808]
  This paper proposes an algorithm and hardware co-design framework that can generate field-programmable gate array (FPGA)-based accelerators for efficient BayesNNs. At the algorithm level, we propose novel multi-exit dropout-based BayesNNs with reduced computational and memory overheads. At the hardware level, this paper introduces a transformation framework that can generate FPGA-based accelerators for the proposed efficient BayesNNs.
  arXiv  Detail & Related papers  (2024-06-20T17:08:42Z)
• A Survey of Serverless Machine Learning Model Inference [0.0]
  Generative AI, Computer Vision, and Natural Language Processing have led to an increased integration of AI models into various products. This survey aims to summarize and categorize the emerging challenges and optimization opportunities for large-scale deep learning serving systems.
  arXiv  Detail & Related papers  (2023-11-22T18:46:05Z)
• Computation-efficient Deep Learning for Computer Vision: A Survey [121.84121397440337]
  Deep learning models have reached or even exceeded human-level performance in a range of visual perception tasks. Deep learning models usually demand significant computational resources, leading to impractical power consumption, latency, or carbon emissions in real-world scenarios. New research focus is computationally efficient deep learning, which strives to achieve satisfactory performance while minimizing the computational cost during inference.
  arXiv  Detail & Related papers  (2023-08-27T03:55:28Z)
• Inducing Gaussian Process Networks [80.40892394020797]
  We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points. The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains. We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-21T05:27:09Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Integrating Deep Learning in Domain Sciences at Exascale [2.241545093375334]
  We evaluate existing packages for their ability to run deep learning models and applications on large-scale HPC systems efficiently. We propose new asynchronous parallelization and optimization techniques for current large-scale heterogeneous systems. We present illustrations and potential solutions for enhancing traditional compute- and data-intensive applications with AI.
  arXiv  Detail & Related papers  (2020-11-23T03:09:58Z)
• Spiking Neural Networks Hardware Implementations and Challenges: a Survey [53.429871539789445]
  Spiking Neural Networks are cognitive algorithms mimicking neuron and synapse operational principles. We present the state of the art of hardware implementations of spiking neural networks. We discuss the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level.
  arXiv  Detail & Related papers  (2020-05-04T13:24:00Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)
• Resource-Efficient Neural Networks for Embedded Systems [23.532396005466627]
  We provide an overview of the current state of the art of machine learning techniques. We focus on resource-efficient inference based on deep neural networks (DNNs), the predominant machine learning models of the past decade. We substantiate our discussion with experiments on well-known benchmark data sets using compression techniques.
  arXiv  Detail & Related papers  (2020-01-07T14:17:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.