Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impacts

• URL: http://arxiv.org/abs/2506.01827v1
• Date: Mon, 02 Jun 2025 16:12:22 GMT
• Title: Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impacts
• Authors: Spencer Banasik,
• Abstract summary: Machine learning algorithms are shown to be an increasingly valuable tool.<n>It is infeasible to run inference with larger models without an accelerator.<n>We aim to improve the LLM speed on a CPU-only environment by modifying the cache architecture.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As machine learning algorithms are shown to be an increasingly valuable tool, the demand for their access has grown accordingly. Oftentimes, it is infeasible to run inference with larger models without an accelerator, which may be unavailable in environments that have constraints such as energy consumption, security, or cost. To increase the availability of these models, we aim to improve the LLM inference speed on a CPU-only environment by modifying the cache architecture. To determine what improvements could be made, we conducted two experiments using Llama.cpp and the QWEN model: running various cache configurations and evaluating their performance, and outputting a trace of the memory footprint. Using these experiments, we investigate the memory access patterns and performance characteristics to identify potential optimizations.

Related papers

• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• Pruning-Based TinyML Optimization of Machine Learning Models for Anomaly Detection in Electric Vehicle Charging Infrastructure [8.29566258132752]
  This paper investigates a pruning method for anomaly detection in resource-constrained environments, specifically targeting EVCI.<n> optimized models achieved significant reductions in model size and inference times, with only a marginal impact on their performance.<n> Notably, our findings indicate that, in the context of EVCI, pruning and FS can enhance computational efficiency while retaining critical anomaly detection capabilities.
  arXiv  Detail & Related papers  (2025-03-19T00:18:37Z)
• The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelines [6.381783966294295]
  Open-source large language models (LLMs) enable developers to create AI-based solutions while maintaining control over aspects such as privacy and compliance. We analyze the performance, particularly the throughput (tokens generated per unit of time) of 20 LLMs using two inference libraries: vLLM and HuggingFace's pipelines.
  arXiv  Detail & Related papers  (2024-08-02T06:56:59Z)
• Online Adaptation of Language Models with a Memory of Amortized Contexts [82.02369596879817]
  Memory of Amortized Contexts (MAC) is an efficient and effective online adaptation framework for large language models. We show how MAC can be combined with and improve the performance of popular alternatives such as retrieval augmented generations.
  arXiv  Detail & Related papers  (2024-03-07T08:34:57Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)
• Optimizing L1 cache for embedded systems through grammatical evolution [1.9371782627708491]
  Grammatical Evolution (GE) is able to efficiently find the best cache configurations for a given set of benchmark applications. Our proposal is able to find cache configurations that obtain an average improvement of $62%$ versus a real world baseline configuration.
  arXiv  Detail & Related papers  (2023-03-06T18:10:00Z)
• Accelerating Deep Learning Classification with Error-controlled Approximate-key Caching [72.50506500576746]
  We propose a novel caching paradigm, that we named approximate-key caching. While approximate cache hits alleviate DL inference workload and increase the system throughput, they however introduce an approximation error. We analytically model our caching system performance for classic LRU and ideal caches, we perform a trace-driven evaluation of the expected performance, and we compare the benefits of our proposed approach with the state-of-the-art similarity caching.
  arXiv  Detail & Related papers  (2021-12-13T13:49:11Z)
• Top-KAST: Top-K Always Sparse Training [50.05611544535801]
  We propose Top-KAST, a method that preserves constant sparsity throughout training. We show that it performs comparably to or better than previous works when training models on the established ImageNet benchmark. In addition to our ImageNet results, we also demonstrate our approach in the domain of language modeling.
  arXiv  Detail & Related papers  (2021-06-07T11:13:05Z)
• Diagonal Memory Optimisation for Machine Learning on Micro-controllers [21.222568055417717]
  Micro controllers and low power CPUs are increasingly being used to perform inference with machine learning models. Small amounts of RAM available on these targets sets limits on size of models which can be executed. diagonal memory optimisation technique is described and shown to achieve memory savings of up to 34.5% when applied to eleven common models.
  arXiv  Detail & Related papers  (2020-10-04T19:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.