Optimizing ML Workload Partitioning between CPUs and CIM Accelerators for Heterogeneous Computing

• URL: http://arxiv.org/abs/2607.05240v1
• Date: Mon, 06 Jul 2026 15:50:44 GMT
• Title: Optimizing ML Workload Partitioning between CPUs and CIM Accelerators for Heterogeneous Computing
• Abstract summary: We propose an ILP-based workload partitioning framework for Computing-in-Memory (CIM) accelerators.<n>It minimizes end-to-end inference latency under RRAM constraints, captures parallelism, and combines empirical profiling with analytical models.<n>Using our framework, heterogeneous CPU-CIM execution speeds up to 30.9x over CPU-only execution on an edge CPU and 7.3x over a high-performance CPU.
• Score: 0.759633586781718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Computing-in-Memory (CIM) accelerators execute Matrix-Vector Multiplications (MVMs) in memory, making them a compelling solution for Machine Learning (ML) workloads. However, existing ML workload partitioning approaches for CIM accelerators do not fully account for Resistive Random Access Memory (RRAM) constraints such as limited memory, high write latency, and limited endurance. They also neglect parallelism, low-level architectural effects, or the Central Processing Unit (CPU) as a complementary compute resource. To address these limitations, we propose an Integer Linear Programming (ILP)-based workload partitioning framework for heterogeneous CPU-CIM systems. It minimizes end-to-end inference latency under RRAM constraints, captures parallelism, and combines empirical profiling with analytical models. Using our framework, heterogeneous CPU-CIM execution achieves speedups of up to 30.9x over CPU-only execution on an edge CPU and 7.3x over a high-performance CPU. A Design Space Exploration (DSE) yields further design insights for future CIM accelerators.

Related papers

• CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution [6.04230666444406]
  CoX-MoE is an Advanced Matrix Extensions (AMX)-enabled CPU- GPU collaborative system.<n>It comprehensively optimize MoE inference by combining coalesced expert execution with strategic workload orchestration for higher throughput.<n>Compared to state-of-the-art frameworks, CoX-MoE delivers significant gains, achieving up to 7.1x and 2.4x higher throughput.
  arXiv  Detail & Related papers  (2026-05-18T05:54:30Z)
• Learning to Share: Selective Memory for Efficient Parallel Agentic Systems [49.78267008828593]
  Agentic systems solve complex tasks by coordinating multiple agents that iteratively reason, invoke tools, and exchange intermediate results.<n>Recent approaches deploy multiple agent teams running in parallel to explore diverse reasoning trajectories.<n>We propose Learning to Share (LTS), a learned shared-memory mechanism for parallel agentic frameworks.
  arXiv  Detail & Related papers  (2026-02-05T18:20:21Z)
• Mixed-Precision Training and Compilation for RRAM-based Computing-in-Memory Accelerators [0.8708298560474775]
  We propose a mixed-precision training and compilation framework for CIM architectures.<n>The biggest challenge is the massive search space, that makes it difficult to find good quantization parameters.<n>In the best case, our approach achieves up to a 2.48x speedup over existing state-of-the-art solutions, with an accuracy loss of only 0.086 %.
  arXiv  Detail & Related papers  (2026-01-29T13:54:55Z)
• An LLVM-Based Optimization Pipeline for SPDZ [0.0]
  We implement a proof-of-concept LLVM-based optimization pipeline for the SPDZ protocol.<n>Our front end accepts a subset of C with lightweight privacy annotations and lowers it to LLVM IR.<n>Our back end performs data-flow and control-flow analysis on the optimized IR to drive a non-blocking runtime scheduler.
  arXiv  Detail & Related papers  (2025-12-11T20:53:35Z)
• A Tensor Compiler for Processing-In-Memory Architectures [8.353569627672622]
  Processing-In-Memory (PIM) devices can accelerate memory-intensive kernels in Machine Learning (ML) models, including Large Language Models (LLMs)<n>Current compilation approaches lack systematic optimization for diverse ML kernels across multiple PIM backends.<n>We design DCC, the first data-centric ML compiler for PIM systems that jointly co-optimizes data rearrangements and compute code optimization.
  arXiv  Detail & Related papers  (2025-11-19T14:58:16Z)
• xLLM Technical Report [57.13120905321185]
  We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework.<n>xLLM builds a novel decoupled service-engine architecture.<n>xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources.
  arXiv  Detail & Related papers  (2025-10-16T13:53:47Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)
• An Experimental Evaluation of Machine Learning Training on a Real Processing-in-Memory System [9.429605859159023]
  Training machine learning (ML) algorithms is a computationally intensive process, which is frequently memory-bound. Memory-centric computing systems, with processing-in-memory capabilities, can alleviate this data movement bottleneck. We implement several representative classic ML algorithms on a real-world general-purpose PIM architecture.
  arXiv  Detail & Related papers  (2022-07-16T09:39:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.