MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration

• URL: http://arxiv.org/abs/2606.17781v1
• Date: Tue, 16 Jun 2026 10:58:02 GMT
• Title: MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration
• Abstract summary: Non-linear vector normalization operations, such as LayerNorm, RMSNorm and Softmax can become critical hardware bottlenecks.<n>Existing accelerators typically implement these functions using dedicated hardware blocks, leading to duplicated resources and inefficient silicon utilization.<n>We propose a Minimalist Vector Engine (MIVE), a programmable architecture capable of executing all three operations within a unified datapath.
• Score: 5.564319540410472
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapid growth of Large Language Models (LLMs) has intensified the need for specialized hardware accelerators that can satisfy stringent inference latency and power constraints. Although matrix multiplications dominate the overall computational workload, non-linear vector normalization operations, such as LayerNorm, RMSNorm and Softmax can become critical hardware bottlenecks. Existing accelerators typically implement these functions using dedicated hardware blocks, leading to duplicated resources and inefficient silicon utilization. To address this limitation, we propose a Minimalist Integer Vector Engine (MIVE), a programmable architecture capable of executing all three operations within a unified datapath. By exploiting common computational patterns across LayerNorm, RMSNorm and Softmax the proposed vector engine maximizes hardware sharing while reducing implementation overhead. Physical ASIC implementation results show that MIVE provides comprehensive multi-function support while achieving higher area and hardware efficiency than most state-of-the-art standalone accelerators.

Related papers

• AutoVecCoder: Teaching LLMs to Generate Explicitly Vectorized Code [69.84692701646769]
  AutoVecCoder is a novel framework designed to empower Large Language Models with the capability of automated explicit vectorization.<n>VecPrompt, an automated data synthesis pipeline, injects domain-specific intrinsic knowledge.<n>VecRL, a reinforcement learning framework, aligns code generation with execution efficiency.<n>AutoVecCoder-8B trained by this framework achieves state-of-the-art performance on the SSE and AVX subsets of SimdBench.
  arXiv  Detail & Related papers  (2026-05-18T07:33:15Z)
• Efficient Matrix Implementation for Rotary Position Embedding [50.45885099010382]
  Rotary Position Embedding (RoPE) has become a core component of modern Transformer architectures across language, vision, and 3D domains.<n>We propose RoME, a mathematically equivalent yet computationally efficient reformulation of RoPE that replaces vector operations with unified matrix transformations.<n> experiments show that RoME delivers substantial acceleration at both the operator and full-model levels.
  arXiv  Detail & Related papers  (2026-04-10T00:17:47Z)
• TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Design [4.251127523213289]
  TriGen is a novel NPU architecture tailored for resource-constrained environments through software-hardware co-design.<n>We show that TriGen achieves an average 2.73x performance speedup and 52% less memory transfer over the baseline NPU design with negligible accuracy loss.
  arXiv  Detail & Related papers  (2026-02-13T14:28:31Z)
• xLLM Technical Report [57.13120905321185]
  We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework.<n>xLLM builds a novel decoupled service-engine architecture.<n>xLLM-Engine co-optimizes system and algorithm designs to fully saturate computing resources.
  arXiv  Detail & Related papers  (2025-10-16T13:53:47Z)
• COMMET: orders-of-magnitude speed-up in finite element method via batch-vectorized neural constitutive updates [0.0]
  We introduce an open-source finite element framework whose architecture has been redesigned to accelerate high-cost updates.<n>FE COMMET features a novel assembly algorithm that supports batched and vectorized evaluations, compute-graph-optimized derivatives that replace automatic differentiation, and distributed-memory parallelism via MPI.
  arXiv  Detail & Related papers  (2025-10-01T13:31:56Z)
• Leveraging ASIC AI Chips for Homomorphic Encryption [12.209134343914537]
  homomorphic encryption (HE) offers strong privacy guarantee, but it requires substantially more resources than computing on plaintext.<n> accelerators have emerged to mitigate this latency issue, but with the high cost of ASICs.<n>We show that HE primitives can be converted to AI operators and accelerated on existing ASIC AI accelerators, like TPUs, which are already widely deployed in the cloud.
  arXiv  Detail & Related papers  (2025-01-13T04:08:14Z)
• SoftmAP: Software-Hardware Co-design for Integer-Only Softmax on Associative Processors [1.8999662338457695]
  Non-linear operators like Softmax and Layernorm remain bottlenecks due to their sensitivity to quantization.<n>We propose SoftmAP, a software-hardware co-design methodology that implements an integer-only low-precision Softmax using In-Memory Compute (IMC) hardware.
  arXiv  Detail & Related papers  (2024-11-26T20:00:54Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts [60.1586169973792]
  M$3$ViT is the latest multi-task ViT model that introduces mixture-of-experts (MoE) MoE achieves better accuracy and over 80% reduction computation but leaves challenges for efficient deployment on FPGA. Our work, dubbed Edge-MoE, solves the challenges to introduce the first end-to-end FPGA accelerator for multi-task ViT with a collection of architectural innovations.
  arXiv  Detail & Related papers  (2023-05-30T02:24:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.