Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling

• URL: http://arxiv.org/abs/2512.22066v1
• Date: Fri, 26 Dec 2025 15:42:29 GMT
• Title: Prefill vs. Decode Bottlenecks: SRAM-Frequency Tradeoffs and the Memory-Bandwidth Ceiling
• Abstract summary: Energy consumption dictates the cost and environmental impact of deploying Large Language Models.<n>This paper investigates the impact of on-chip size and operating frequency on the energy efficiency and performance of LLM inference.
• Score: 5.606289163036201
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Energy consumption dictates the cost and environmental impact of deploying Large Language Models. This paper investigates the impact of on-chip SRAM size and operating frequency on the energy efficiency and performance of LLM inference, focusing on the distinct behaviors of the compute-bound prefill and memory-bound decode phases. Our simulation methodology combines OpenRAM for energy modeling, LLMCompass for latency simulation, and ScaleSIM for systolic array operational intensity. Our findings show that total energy use is predominantly determined by SRAM size in both phases, with larger buffers significantly increasing static energy due to leakage, which is not offset by corresponding latency benefits. We quantitatively explore the memory-bandwidth bottleneck, demonstrating that while high operating frequencies reduce prefill latency, their positive impact on memory-bound decode latency is capped by the external memory bandwidth. Counter-intuitively, high compute frequency can reduce total energy by reducing execution time and consequently decreasing static energy consumption more than the resulting dynamic power increase. We identify an optimal hardware configuration for the simulated workload: high operating frequencies (1200MHz-1400MHz) and a small local buffer size of 32KB to 64KB. This combination achieves the best energy-delay product, balancing low latency with high energy efficiency. Furthermore, we demonstrate how memory bandwidth acts as a performance ceiling, and that increasing compute frequency only yields performance gains up to the point where the workload becomes memory-bound. This analysis provides concrete architectural insights for designing energy-efficient LLM accelerators, especially for datacenters aiming to minimize their energy overhead.

Related papers

• LLM Inference in a Flash! [66.43896490006965]
  Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks.<n>We present an end-to-end integer-only quantization approach to eliminate expensive floating-point computations.
  arXiv  Detail & Related papers  (2026-09-14T18:04:30Z)
• Understand and Accelerate Memory Processing Pipeline for Disaggregated LLM Inference [45.78442013449325]
  Large language models rely on efficient long-context processing and generation mechanisms to support complex reasoning.<n>We show that these optimizations can be unified into a four-step memory processing pipeline: Prepare Memory, Compute Relevancy, Retrieval, and Apply to Inference.<n>Motivated by this insight, we argue that textbfheterogeneous systems are well-suited to accelerate memory processing and thus end-to-end inference.
  arXiv  Detail & Related papers  (2026-03-30T21:03:39Z)
• Neural Dynamics Self-Attention for Spiking Transformers [59.57278823240336]
  Spiking Neural Networks (SNNs) with Transformer architectures offer a promising pathway to balance energy efficiency and performance.<n>Existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference.<n>We propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements.
  arXiv  Detail & Related papers  (2026-03-09T12:23:37Z)
• RPU -- A Reasoning Processing Unit [4.783828820539779]
  Reasoning Processing Unit (RPU) is a chiplet-based architecture designed to address the challenges of the modern memory wall.<n>RPU performs up to 45.3x lower latency and 18.6x higher throughput over an H100 system at ISO-TDP on Llama3-405B.
  arXiv  Detail & Related papers  (2026-02-20T19:13:19Z)
• DAF: An Efficient End-to-End Dynamic Activation Framework for on-Device DNN Training [41.09085549544767]
  We introduce a Dynamic Activation Framework (DAF) that enables scalable and efficient on-device training through system-level optimizations.<n>DAF achieves both memory- and time-efficient dynamic quantization training by addressing key system bottlenecks.<n> Evaluations on various deep learning models across embedded and mobile platforms demonstrate up to a $22.9times$ reduction in memory usage and a $3.2times$ speedup.
  arXiv  Detail & Related papers  (2025-07-09T08:59:30Z)
• Benchmarking of CPU-intensive Stream Data Processing in The Edge Computing Systems [41.19058376513831]
  This paper evaluates the power consumption and performance characteristics of a single processing node within an edge cluster using a synthetic microbenchmark.<n>Results show how an optimal measure can lead to optimized usage of edge resources, given both performance and power consumption.
  arXiv  Detail & Related papers  (2025-05-12T17:02:02Z)
• CalibQuant: 1-Bit KV Cache Quantization for Multimodal LLMs [45.77132019859689]
  CalibQuant is a visual quantization strategy that drastically reduces both memory and computational overhead.<n>We achieve a 10x throughput increase on InternVL models.
  arXiv  Detail & Related papers  (2025-02-15T05:08:01Z)
• MEADOW: Memory-efficient Dataflow and Data Packing for Low Power Edge LLMs [5.88896081401217]
  We introduce MEADOW, a framework that significantly reduces the off-chip memory access for large language models.<n> MEADOW demonstrates 1.5x and 2.5x lower decode and prefill latency, respectively, compared to a GEMM-based LLM implementation.<n> MEADOW achieves an end-to-end latency improvement of over 40%, compared to prior LLM optimization works.
  arXiv  Detail & Related papers  (2025-02-14T23:50:37Z)
• Fast Matrix Multiplications for Lookup Table-Quantized LLMs [58.11584672945781]
  FLUTE is a flexible lookup table engine for LUT-quantized LLMs.<n>At batch sizes 32 and quantization group size of 128, the FLUTE kernel can be 2-4x faster than existing GEMM kernels.
  arXiv  Detail & Related papers  (2024-07-15T17:55:42Z)
• OPIMA: Optical Processing-In-Memory for Convolutional Neural Network Acceleration [5.0389804644646174]
  We introduce OPIMA, a processing-in-memory (PIM)-based machine learning accelerator. PIM struggles to achieve high throughput and energy efficiency due to internal data movement bottlenecks. We show that OPIMA can achieve 2.98x higher throughput and 137x better energy efficiency than the best-known prior work.
  arXiv  Detail & Related papers  (2024-07-11T06:12:04Z)
• LPViT: Low-Power Semi-structured Pruning for Vision Transformers [43.126752035656196]
  Vision transformers have emerged as a promising alternative to convolutional neural networks for image analysis tasks.<n>One significant drawback of ViTs is their resource-intensive nature, leading to increased memory footprint, complexity, and power consumption.<n>We introduce a new block-structured pruning to address the resource-intensive issue for ViTs, offering a balanced trade-off between accuracy and hardware acceleration.
  arXiv  Detail & Related papers  (2024-07-02T08:58:19Z)
• DAISM: Digital Approximate In-SRAM Multiplier-based Accelerator for DNN Training and Inference [4.718504401468233]
  PIM solutions rely either on novel memory technologies that have yet to mature or bit-serial computations that have significant performance overhead and scalability issues. Our work proposes an in-SRAM digital multiplier, that uses a conventional memory to perform bit-parallel computations, leveraging multiple wordlines activation. We then introduce DAISM, an architecture leveraging this multiplier, which achieves up to two orders of magnitude higher area efficiency compared to the SOTA counterparts, with competitive energy efficiency.
  arXiv  Detail & Related papers  (2023-05-12T10:58:21Z)
• Hitless memory-reconfigurable photonic reservoir computing architecture [1.4479776639062198]
  Reservoir computing is an analog bio-inspired computation model for efficiently processing time-dependent signals. We propose a novel TDRC architecture based on an asymmetric Mach-Zehnder interferometer integrated in a resonant cavity. We demonstrate this approach on the temporal bitwise XOR task and conclude that this way of memory capacity reconfiguration allows optimal performance to be achieved.
  arXiv  Detail & Related papers  (2022-07-13T14:43:40Z)
• Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution [44.909233016062906]
  We build a memory-efficient image super-resolution network (FMEN) for resource-constrained devices. FMEN runs 33% faster and reduces 74% memory consumption compared with the state-of-the-art EISR model: E-RFDN. FMEN-S achieves the lowest memory consumption and the second shortest runtime in NTIRE 2022 challenge on efficient super-resolution.
  arXiv  Detail & Related papers  (2022-04-18T16:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.