PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers

• URL: http://arxiv.org/abs/2411.19114v1
• Date: Thu, 28 Nov 2024 13:02:41 GMT
• Title: PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers
• Authors: Gwangoo Yeo, Jiin Kim, Yujeong Choi, Minsoo Rhu,
• Abstract summary: PREBA is a hardware/software co-design targeting MIG inference servers. It provides a 3.7x throughput, 3.4x reduction in tail latency, 3.5x improvement in energy-efficiency and 3.0x improvement in cost-efficiency.
• Score: 3.0518650058744075
• License:
• Abstract: NVIDIA's Multi-Instance GPU (MIG) is a feature that enables system designers to reconfigure one large GPU into multiple smaller GPU slices. This work characterizes this emerging GPU and evaluates its effectiveness in designing high-performance AI inference servers. Our study reveals that the data preprocessing stage of AI inference causes significant performance bottlenecks to MIG. To this end, we present PREBA, which is a hardware/software co-design targeting MIG inference servers. Our first proposition is an FPGA-based data preprocessing accelerator that unlocks the full potential of MIG with domain-specific acceleration of data preprocessing. The MIG inference server unleashed from preprocessing overheads is then augmented with our dynamic batching system that enables high-performance inference. PREBA is implemented end-to-end in real systems, providing a 3.7x improvement in throughput, 3.4x reduction in tail latency, 3.5x improvement in energy-efficiency, and 3.0x improvement in cost-efficiency.

Related papers

• Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inference [20.68731158617374]
  Dovetail is an approach that deploys the draft model on the GPU to generate draft tokens while allowing the target model to perform parallel verification on the CPU. Dovetail achieves an inference speed of 5.86 tokens per second for LLaMA2-Chat-7B using 3GB of VRAM, representing an approximately 2.77x improvement over CPU-only inference.
  arXiv  Detail & Related papers  (2024-12-25T15:45:18Z)
• Pushing the Performance Envelope of DNN-based Recommendation Systems Inference on GPUs [13.720423381263409]
  We show that the embedding stage continues to be the primary bottleneck in the GPU inference pipeline, leading to a 3.2x embedding-only performance slowdown. We propose specialized plug-and-play-based software prefetching and L2 pinning techniques, which help in hiding and decreasing the latencies. Our proposed techniques improve performance by up to 103% for the embedding stage, and up to 77% for the overall DLRM inference pipeline.
  arXiv  Detail & Related papers  (2024-10-29T17:13:54Z)
• Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading [2.8231000588510757]
  Transformers and large language models(LLMs) have seen rapid adoption in all domains. Training of transformers is very expensive and often hits a memory wall'' We propose a novel technique to split the LLM into subgroups, whose update phase is scheduled on either the CPU or the GPU.
  arXiv  Detail & Related papers  (2024-10-26T00:43:59Z)
• Towards Deterministic End-to-end Latency for Medical AI Systems in NVIDIA Holoscan [0.35516599670943777]
  Medical device manufacturers are keen to maximize the advantages afforded by AI and ML by consolidating multiple applications onto a single platform. concurrent execution of several AI applications, each with its own visualization components, leads to unpredictable end-to-end latency. This paper addresses these challenges within the context of the Holoscan platform, a real-time AI system for streaming sensor data and images.
  arXiv  Detail & Related papers  (2024-02-06T23:20:34Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing [66.00729477511219]
  Given a function represented as a computation graph, traditional architectures face challenges in efficiently computing its nth-order gradient. We introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture. We present results that demonstrate 1.8-4.8x and 1.5-3.6x speedup compared to CPU and GPU baselines respectively.
  arXiv  Detail & Related papers  (2023-08-11T04:24:39Z)
• MAPLE-X: Latency Prediction with Explicit Microprocessor Prior Knowledge [87.41163540910854]
  Deep neural network (DNN) latency characterization is a time-consuming process. We propose MAPLE-X which extends MAPLE by incorporating explicit prior knowledge of hardware devices and DNN architecture latency.
  arXiv  Detail & Related papers  (2022-05-25T11:08:20Z)
• PARIS and ELSA: An Elastic Scheduling Algorithm for Reconfigurable Multi-GPU Inference Servers [0.9854614058492648]
  NVIDIA's Ampere GPU architecture provides features to "reconfigure" one large, monolithic GPU into multiple smaller "GPU partitions" In this paper, we study this emerging GPU architecture with reconfigurability to develop a high-performance multi-GPU ML inference server.
  arXiv  Detail & Related papers  (2022-02-27T23:30:55Z)
• MAPLE: Microprocessor A Priori for Latency Estimation [81.91509153539566]
  Modern deep neural networks must demonstrate state-of-the-art accuracy while exhibiting low latency and energy consumption. Measuring the latency of every evaluated architecture adds a significant amount of time to the NAS process. We propose Microprocessor A Priori for Estimation Estimation MAPLE that does not rely on transfer learning or domain adaptation.
  arXiv  Detail & Related papers  (2021-11-30T03:52:15Z)
• Multi-Exit Semantic Segmentation Networks [78.44441236864057]
  We propose a framework for converting state-of-the-art segmentation models to MESS networks. specially trained CNNs that employ parametrised early exits along their depth to save during inference on easier samples. We co-optimise the number, placement and architecture of the attached segmentation heads, along with the exit policy, to adapt to the device capabilities and application-specific requirements.
  arXiv  Detail & Related papers  (2021-06-07T11:37:03Z)
• Faster than FAST: GPU-Accelerated Frontend for High-Speed VIO [46.20949184826173]
  This work focuses on the applicability of efficient low-level, GPU hardware-specific instructions to improve on existing computer vision algorithms. Especially non-maxima suppression and the subsequent feature selection are prominent contributors to the overall image processing latency.
  arXiv  Detail & Related papers  (2020-03-30T14:16:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.