LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi

• URL: http://arxiv.org/abs/2504.02118v1
• Date: Wed, 02 Apr 2025 20:29:39 GMT
• Title: LLMPi: Optimizing LLMs for High-Throughput on Raspberry Pi
• Authors: Mahsa Ardakani, Jinendra Malekar, Ramtin Zand,
• Abstract summary: Large Language Models (LLMs) on resource-constrained edge devices like the Raspberry Pi presents challenges in computational efficiency, power consumption, and response latency.<n>This paper explores quantization-based optimization techniques to enable high- throughput, energy-efficient execution of LLMs on low-power embedded systems.<n>Our findings highlight the potential of quantized LLMs for real-time conversational AI on edge devices, paving the way for low-power, high-efficiency AI deployment in mobile and embedded applications.
• Score: 0.48212500317840945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying Large Language Models (LLMs) on resource-constrained edge devices like the Raspberry Pi presents challenges in computational efficiency, power consumption, and response latency. This paper explores quantization-based optimization techniques to enable high-throughput, energy-efficient execution of LLMs on low-power embedded systems. Our approach leverages k-quantization, a Post-Training Quantization (PTQ) method designed for different bit-widths, enabling efficient 2-bit, 4-bit, 6-bit, and 8-bit weight quantization. Additionally, we employ ternary quantization using Quantization-Aware Training (QAT) for BitNet models, allowing for more effective adaptation to lower-bit representations while preserving accuracy. Our findings highlight the potential of quantized LLMs for real-time conversational AI on edge devices, paving the way for low-power, high-efficiency AI deployment in mobile and embedded applications. This study demonstrates that aggressive quantization strategies can significantly reduce energy consumption while maintaining inference quality, making LLMs practical for resource-limited environments.

Related papers

• Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency [6.306413686006502]
  We conduct a comprehensive analysis of 28 quantized Large Language Models (LLMs) from the Ollama library. We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings.
  arXiv  Detail & Related papers  (2025-04-04T11:29:30Z)
• QuartDepth: Post-Training Quantization for Real-Time Depth Estimation on the Edge [55.75103034526652]
  We propose QuartDepth which adopts post-training quantization to quantize MDE models with hardware accelerations for ASICs. Our approach involves quantizing both weights and activations to 4-bit precision, reducing the model size and computation cost. We design a flexible and programmable hardware accelerator by supporting kernel fusion and customized instruction programmability.
  arXiv  Detail & Related papers  (2025-03-20T21:03:10Z)
• Quantizing Large Language Models for Code Generation: A Differentiated Replication [51.85505914274633]
  Large Language Models (LLMs) have shown an impressive capability in code generation and, specifically, to automatically implement requirements described in natural language.<n>LLMs pose significant challenges related to their memory (and, consequently, carbon) footprint.<n>New frontier for LLM quantization is 4-bit precision, resulting in an average memory footprint reduction of 70%.
  arXiv  Detail & Related papers  (2025-03-10T09:26:08Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE) RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers. Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• Explore Activation Sparsity in Recurrent LLMs for Energy-Efficient Neuromorphic Computing [3.379854610429579]
  Recurrent Large Language Models (R-LLM) have proven effective in mitigating the complexity of self-attention. We propose a low-cost, training-free algorithm to sparsify R-LLMs' activations to enhance energy efficiency on neuromorphic hardware.
  arXiv  Detail & Related papers  (2025-01-09T19:13:03Z)
• A Comprehensive Study on Quantization Techniques for Large Language Models [0.0]
  Large Language Models (LLMs) have been extensively researched and used in both academia and industry. LLMs present significant challenges for deployment on resource-constrained IoT devices and embedded systems. Quantization, a technique that reduces the precision of model values to a smaller set of discrete values, offers a promising solution.
  arXiv  Detail & Related papers  (2024-10-30T04:55:26Z)
• AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer [54.713778961605115]
  Vision Transformer (ViT) has become one of the most prevailing fundamental backbone networks in the computer vision community. We propose a novel non-uniform quantizer, dubbed the Adaptive Logarithm AdaLog (AdaLog) quantizer.
  arXiv  Detail & Related papers  (2024-07-17T18:38:48Z)
• Q-Sparse: All Large Language Models can be Fully Sparsely-Activated [93.45300714803429]
  We introduce Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs) Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference. We also introduce Block Q-Sparse for batch training and inference.
  arXiv  Detail & Related papers  (2024-07-15T17:59:29Z)
• Low-Rank Quantization-Aware Training for LLMs [8.535254310145005]
  Large language models (LLMs) are omnipresent, however their practical deployment is challenging due to their ever increasing computational and memory demands. We propose LR-QAT -- a lightweight and memory-efficient QAT algorithm for LLMs. Our method outperforms common post-training quantization (PTQ) approaches and reaches the same model performance as full-model QAT at the fraction of its memory usage.
  arXiv  Detail & Related papers  (2024-06-10T15:44:22Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization [27.79783067245817]
  Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. This paper presents Efficient Adaptation and Quantization-aware (PEQA) - a simple yet effective method that combines the advantages of PEFT with quantized LLMs.
  arXiv  Detail & Related papers  (2023-05-23T15:20:01Z)
• Neural Network Quantization with AI Model Efficiency Toolkit (AIMET) [15.439669159557253]
  We present an overview of neural network quantization using AI Model Efficiency Toolkit (AIMET) AIMET is a library of state-of-the-art quantization and compression algorithms designed to ease the effort required for model optimization. We provide a practical guide to quantization via AIMET by covering PTQ and QAT, code examples and practical tips.
  arXiv  Detail & Related papers  (2022-01-20T20:35:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.