DPU or GPU for Accelerating Neural Networks Inference -- Why not both? Split CNN Inference

• URL: http://arxiv.org/abs/2605.00174v1
• Date: Thu, 30 Apr 2026 19:49:55 GMT
• Title: DPU or GPU for Accelerating Neural Networks Inference -- Why not both? Split CNN Inference
• Abstract summary: partitioning CNN inference across DPU and GPU (Split CNN Inference) is proposed.<n>GNN-based partition index prediction method is proposed to automate the partitioning of CNNs needed for the Split Inference.<n>Results demonstrate up to 2.48x latency improvement over DPU-only execution and up to 3.37x over GPU-only execution.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video and image streaming on edge devices requires low latency. To address this, Neural Networks (NNs) are widely used, and prior work mainly focuses on accelerating them with single hardware units such as Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), and Deep Learning Processing Units (DPUs). However, further reductions in latency can be observed by combining these units. In this paper, partitioning CNN inference across DPU and GPU (Split CNN Inference) is proposed. The first partition runs on the AI engines (DPU) of a Versal VCK190, which consists of initial CNN layers processing the input images. The DPU processes the first partition near the source of the data. Pipelined asynchronously, a GPU runs the remaining layers. The GPU (NVIDIA RTX 2080) processes the second partition, albeit having reduced the data transfer between the data source (storage/camera) and the GPU. Furthermore, a Graph Neural Network (GNN)-based partition index prediction method is proposed to automate the partitioning of CNNs needed for the Split Inference. Well established models such as LeNet-5, ResNet18/50/101/152, VGG16, and MobileNetv2 are analyzed. Results demonstrate up to 2.48x latency improvement over DPU-only execution and up to 3.37x over GPU-only execution. The trained GNN model splits the layers between the appropriate devices with 96.27% accuracy.

Related papers

• Minute-Long Videos with Dual Parallelisms [57.22737565366549]
  Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos.<n>We propose a novel distributed inference strategy, termed DualParal.<n>Instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs.
  arXiv  Detail & Related papers  (2025-05-27T11:55:22Z)
• Distributed Convolutional Neural Network Training on Mobile and Edge Clusters [0.9421843976231371]
  Recent efforts have emerged to localize machine learning tasks fully on the edge. This brings advantages in reduced latency and increased privacy, but necessitates working with resource-constrained devices. We describe an approach for distributed CNN training exclusively on mobile and edge devices.
  arXiv  Detail & Related papers  (2024-09-11T02:44:28Z)
• Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awareness [4.8412870364335925]
  Quiver is a distributed GPU-based GNN serving system with low-latency and high- throughput. We show that Quiver achieves up to 35 times lower latency with an 8 times higher throughput compared to state-of-the-art GNN approaches.
  arXiv  Detail & Related papers  (2023-05-18T10:34:23Z)
• A Study on the Intersection of GPU Utilization and CNN Inference [8.084016058894779]
  We show that there is room to improve the inference-time GPU utilization of convolutional neural network (CNN) inference. Our study makes the case that there is room to improve the inference-time GPU utilization of CNNs and that knowledge of GPU utilization has the potential to benefit even applications that do not target utilization itself.
  arXiv  Detail & Related papers  (2022-12-15T16:11:40Z)
• Dynamic Split Computing for Efficient Deep Edge Intelligence [78.4233915447056]
  We introduce dynamic split computing, where the optimal split location is dynamically selected based on the state of the communication channel. We show that dynamic split computing achieves faster inference in edge computing environments where the data rate and server load vary over time.
  arXiv  Detail & Related papers  (2022-05-23T12:35:18Z)
• DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks [58.48833325238537]
  Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. In this paper, we presentGNN that optimize the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters. Our results on four common GNN benchmark datasets show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets.
  arXiv  Detail & Related papers  (2021-04-14T08:46:35Z)
• L2PF -- Learning to Prune Faster [57.32153461504626]
  We present a multi-task, try-and-learn method, discretely learning redundant filters of the CNN and a continuous action of how long the layers have to be fine-tuned. For ResNet20, we have achieved a compression ratio of 3.84 x with minimal accuracy degradation. Compared to the state-of-the-art pruning method, we reduced the GPU hours by 1.71 x.
  arXiv  Detail & Related papers  (2021-01-07T18:13:37Z)
• Performance Prediction for Convolutional Neural Networks in Edge Devices [0.8602553195689513]
  Convolutional Neural Network (CNN) based applications on edge devices near the source of data can meet the latency and privacy challenges. We present and compare five (5) of the widely used Machine Learning based methods for execution time prediction of CNNs on two (2) edge GPU platforms.
  arXiv  Detail & Related papers  (2020-10-21T20:21:25Z)
• RT3D: Achieving Real-Time Execution of 3D Convolutional Neural Networks on Mobile Devices [57.877112704841366]
  This paper proposes RT3D, a model compression and mobile acceleration framework for 3D CNNs. For the first time, real-time execution of 3D CNNs is achieved on off-the-shelf mobiles.
  arXiv  Detail & Related papers  (2020-07-20T02:05:32Z)
• TFApprox: Towards a Fast Emulation of DNN Approximate Hardware Accelerators on GPU [0.4817429789586127]
  Energy efficiency of hardware accelerators of deep neural networks (DNN) can be improved by introducing approximate arithmetic circuits. A software emulation of the DNN accelerator is usually executed on CPU or GPU. This emulation is typically two or three orders of magnitude slower than a software DNN implementation running on or emulated.
  arXiv  Detail & Related papers  (2020-02-21T08:22:56Z)
• Efficient Video Semantic Segmentation with Labels Propagation and Refinement [138.55845680523908]
  This paper tackles the problem of real-time semantic segmentation of high definition videos using a hybrid GPU / CPU approach. We propose an Efficient Video(EVS) pipeline that combines: (i) On the CPU, a very fast optical flow method, that is used to exploit the temporal aspect of the video and propagate semantic information from one frame to the next. On the popular Cityscapes dataset with high resolution frames (2048 x 1024), the proposed operating points range from 80 to 1000 Hz on a single GPU and CPU.
  arXiv  Detail & Related papers  (2019-12-26T11:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.