ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs

• URL: http://arxiv.org/abs/2603.07770v1
• Date: Sun, 08 Mar 2026 19:20:25 GMT
• Title: ArcLight: A Lightweight LLM Inference Architecture for Many-Core CPUs
• Abstract summary: ArcLight is a lightweight LLM inference architecture designed from the ground up for many-core CPUs.<n>We show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput.
• Score: 50.29619653515229
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although existing frameworks for large language model (LLM) inference on CPUs are mature, they fail to fully exploit the computation potential of many-core CPU platforms. Many-core CPUs are widely deployed in web servers and high-end networking devices, and are typically organized into multiple NUMA nodes that group cores and memory. Current frameworks largely overlook the substantial overhead of cross-NUMA memory access, limiting inference scalability and intelligence enabling on such platforms. To address this limitation, we build ArcLight, a lightweight LLM inference architecture designed from the ground up for many-core CPUs. ArcLight integrates efficient memory management and thread scheduling, and introduces finely controlled tensor parallelism to mitigate the cross-node memory access wall. Experimental results show that ArcLight significantly surpasses the performance ceiling of mainstream frameworks, achieving up to 46% higher inference throughput. Moreover, ArcLight maintains compatibility with arbitrary CPU devices. ArcLight is publicly available at https://github.com/OpenBMB/ArcLight.

Related papers

• CodegenBench: Can LLMs Write Efficient Code Across Architectures? [17.06172900876354]
  Large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments.<n>CodegenBench is a benchmark suite designed to evaluate the generation of efficient parallel code across three distinct hardware platforms.
  arXiv  Detail & Related papers  (2026-06-01T12:55:10Z)
• Blink: CPU-Free LLM Inference by Delegating the Serving Stack to GPU and SmartNIC [1.1476836276366738]
  Large Language Model (LLM) inference is rapidly becoming a core datacenter service.<n>Current serving stacks keep the host on the critical path for orchestration and token-level control.<n>We introduce Blink, an end-to-end serving architecture that removes the host CPU from the steady-state inference path.
  arXiv  Detail & Related papers  (2026-04-08T21:27:47Z)
• HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures [20.525243835887558]
  SuperNode represents data movement using cache operators within the compiler.<n>We implement SuperNode within the production deep learning framework MindSpore.<n>We show that SuperNode reduces peak device memory usage by up to 26% for inference while maintaining end-to-end performance.
  arXiv  Detail & Related papers  (2026-01-31T14:29:13Z)
• LightMem: Lightweight and Efficient Memory-Augmented Generation [72.21680105265824]
  We introduce a new memory system called LightMem, which strikes a balance between the performance and efficiency of memory systems.<n>Inspired by the Atkinson-Shiffrin model of human memory, LightMem organizes memory into three complementary stages.<n>Experiments on LongMemEval with GPT and Qwen backbones show that LightMem outperforms strong baselines in accuracy (up to 10.9% gains) while reducing token usage by up to 117x.
  arXiv  Detail & Related papers  (2025-10-21T17:58:17Z)
• SparAMX: Accelerating Compressed LLMs Token Generation on AMX-powered CPUs [5.760049762453579]
  Accelerating large language models with CPUs enables broader AI access at a lower cost and power consumption.<n>We provide a set of open-source customized sparse kernels that can speed up any PyTorch model.<n>We demonstrate for the first time the use of unstructured sparsity in the attention achieving a $1.14 times$ speedup over the current systems.
  arXiv  Detail & Related papers  (2025-02-18T02:26:34Z)
• MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs [55.95879347182669]
  MoE architecture is renowned for its ability to increase model capacity without a proportional increase in inference cost. MoE-Lightning introduces a novel CPU-GPU-I/O pipelining schedule, CGOPipe, with paged weights to achieve high resource utilization. MoE-Lightning can achieve up to 10.3x higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB)
  arXiv  Detail & Related papers  (2024-11-18T01:06:12Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• Efficient LLM Inference on CPUs [8.802223672775844]
  Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. deploying these models has been challenging due to the astronomical amount of model parameters. We propose an effective approach that can make the deployment of LLMs more efficiently.
  arXiv  Detail & Related papers  (2023-11-01T13:08:50Z)
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing [66.00729477511219]
  Given a function represented as a computation graph, traditional architectures face challenges in efficiently computing its nth-order gradient. We introduce INR-Arch, a framework that transforms the computation graph of an nth-order gradient into a hardware-optimized dataflow architecture. We present results that demonstrate 1.8-4.8x and 1.5-3.6x speedup compared to CPU and GPU baselines respectively.
  arXiv  Detail & Related papers  (2023-08-11T04:24:39Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.