Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs

• URL: http://arxiv.org/abs/2609.04168v1
• Date: Thu, 03 Sep 2026 17:53:44 GMT
• Title: Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs
• Abstract summary: Para-Pipe is a hierarchical mapping framework that integrates intra- and inter-stage operator parallelism within a pipelined architecture.<n>Para-Pipe navigates the trade-off between throughput and latency by selectively fine-tuning parallelism levels within and across pipeline stages.
• Score: 22.6523068033902
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges. Traditional pipelining techniques distributing the computation across different on-chip processing units, while effective for throughput, do not address the latency demands posed by modern neural networks with complex interdependencies and extensive operator parallelism. There is a potential in leveraging operator parallelism to enable concurrent execution across multiple processing units, thereby reducing inference latency. However, prioritizing pipelining or parallel execution often necessitates a compromise, where optimizing one performance metric adversely impacts the other. This paper introduces Para-Pipe, a hierarchical mapping framework that integrates intra- and inter-stage operator parallelism within a pipelined architecture. Para-Pipe navigates the trade-off between throughput and latency by selectively fine-tuning parallelism levels within and across pipeline stages. This strategy can significantly reduce inter-processor communication overhead, significantly improving energy efficiency. Our evaluation demonstrates that Para-Pipe generates multiple Pareto-optimal configurations, achieving a balance between throughput and latency on an Amlogic SoC equipped with ARM big.LITTLE CPUs and GPU, as well as the Black Sesame Technology SoC featuring a deep learning accelerator and two DSPs. More importantly, throughput-optimized configurations under Para-Pipe on Amlogic SoC show an average energy efficiency improvement of 11.0% over purely pipelined strategies and 23.3% relative to non-pipelined parallel execution.

Related papers

• CTA-Pipelining: A Latency-Oriented Spatial Scaling Method for Multi-GPU Systems [2.870762512009438]
  We introduce CTA-pipelining, an execution paradigm designed to exploit shared-memory multi-GPU systems.<n>As a latency-oriented spatial scaling technique, CTA-pipelining leverages the Cooperative Thread Array level.<n>Results show CTA-pipelining reduces latency by up to 31.8% compared to micro-batching.
  arXiv  Detail & Related papers  (2026-07-08T18:54:08Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks [6.447439020088275]
  Breakthroughs in AI have fueled an explosion of large language model (LLM)-powered applications.<n>Dense models continue to dominate because of their strong ability, scalability, ease of fine-tuning, and versatility across diverse tasks.<n>This paper investigates the workloads of two representative dense LLMs, focusing in particular on intra-node parallelization schemes.
  arXiv  Detail & Related papers  (2026-03-05T21:33:24Z)
• Orchestrating Multimodal DNN Workloads in Wireless Neural Processing [57.510786937781866]
  In edge inference, wireless resource allocation and accelerator deep neural computation (DNN) scheduling have yet to be co-optimized in an end-to-end manner.<n>This paper investigates a paradigm that integrates wireless transmission and multi-core execution into a unified end-to-end pipeline.
  arXiv  Detail & Related papers  (2026-03-02T17:25:43Z)
• GANGR: GAN-Assisted Scalable and Efficient Global Routing Parallelization [0.6117371161379208]
  Global routing is a critical stage in electronic design automation (EDA)<n>This paper introduces Wasserstein generative networks (WGANs) to enable more effective parallelization.<n>The proposed algorithm is tested on the latest ISPD'24 contest benchmarks, demonstrating up to 40% reduction with only 0.002% degradation in routing quality as compared to state-of-the-art routers.
  arXiv  Detail & Related papers  (2025-11-21T00:32:33Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• Vectorized Online POMDP Planning [4.097364225798782]
  POMDP is a framework for planning under partial observability problems.<n>We propose Vectorized Online POMDP Planner (VOPP), a novel parallel online solver.<n>VOPP represents all data structures related to planning as a collection of tensors and implements all planning steps as fully vectorized computations.
  arXiv  Detail & Related papers  (2025-10-31T05:21:39Z)
• Intra-DP: A High Performance Collaborative Inference System for Mobile Edge Computing [67.98609858326951]
  Intra-DP is a high-performance collaborative inference system optimized for deep neural networks (DNNs) on mobile devices.<n>It reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines.<n>The evaluation demonstrates that Intra-DP reduces per-inference latency by up to 50% and energy consumption by up to 75% compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-07-08T09:50:57Z)
• APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs [81.5049387116454]
  We introduce APB, an efficient long-context inference framework.<n>APB uses multi-host approximate attention to enhance prefill speed.<n>APB achieves speeds of up to 9.2x, 4.2x, and 1.6x compared with FlashAttn, RingAttn, and StarAttn, respectively.
  arXiv  Detail & Related papers  (2025-02-17T17:59:56Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.