IterL2Norm: Fast Iterative L2-Normalization

• URL: http://arxiv.org/abs/2412.04778v2
• Date: Fri, 17 Jan 2025 08:58:17 GMT
• Title: IterL2Norm: Fast Iterative L2-Normalization
• Abstract summary: Transformer-based large language models are a memory-bound model whose operation is based on a large amount of data that are marginally reused.<n> Layer normalization is one of the key workloads in the transformer model, following each of multi-head attention and feed-forward network blocks.<n>We introduce an iterative L2-normalization method for 1D input (IterL2Norm), ensuring fast convergence to the steady-state solution within five iteration steps.
• Score: 1.4602363426887837
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based large language models are a memory-bound model whose operation is based on a large amount of data that are marginally reused. Thus, the data movement between a host and accelerator likely dictates the total wall-clock time. Layer normalization is one of the key workloads in the transformer model, following each of multi-head attention and feed-forward network blocks. To reduce data movement, layer normalization needs to be performed on the same chip as the matrix-matrix multiplication engine. To this end, we introduce an iterative L2-normalization method for 1D input (IterL2Norm), ensuring fast convergence to the steady-state solution within five iteration steps and high precision, outperforming the fast inverse square root algorithm in six out of nine cases for FP32 and five out of nine for BFloat16 across the embedding lengths used in the OPT models. Implemented in 32/28nm CMOS, the IterL2Norm macro normalizes $d$-dimensional vectors, where $64 \leq d \leq 1024$, with a latency of 116-227 cycles at 100MHz/1.05V.

Related papers

• ELiTeFormer: An Efficient Transformer for FPGAs [3.884894816711594]
  Transformer blocks are prevalent in large language model (LLM) but present deployment challenges.<n>We present ELiTeFormer, the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections.
  arXiv  Detail & Related papers  (2026-07-04T00:52:06Z)
• Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders [123.58723804218151]
  Vision foundation models are bottlenecked by the quadratic cost of self-attention.<n>We present C-GSPN based on 2D spatial propagation propagation.
  arXiv  Detail & Related papers  (2026-05-30T14:29:43Z)
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding [48.9323408950142]
  We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations.<n>We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-gressive recurrent updates under mild assumptions.
  arXiv  Detail & Related papers  (2026-04-23T02:12:58Z)
• Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall [0.0]
  Fan Duality Model resolves tension between memory efficiency and associative recall in sequence modeling.<n>FDM separates sequence processing into two components: a wave component that compresses long-range patterns into a hidden state, and a particle component that retrieves specific tokens.<n>We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimize the cache jointly with embeddings.
  arXiv  Detail & Related papers  (2026-04-09T02:00:30Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• GSPN-2: Efficient Parallel Sequence Modeling [101.33780567131716]
  Generalized Spatial Propagation Network (GSPN) addresses this by replacing quadratic self-attention with a line-scan propagation scheme.<n>GSPN-2 establishes a new efficiency frontier for modeling global spatial context in vision applications.
  arXiv  Detail & Related papers  (2025-11-28T07:26:45Z)
• Orthogonal Finetuning Made Scalable [87.49040247077389]
  Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deployment.<n>We identify the core computational bottleneck in OFT as its weight-centric implementation, which relies on costly matrix-matrix multiplications with cubic complexity.<n>We propose OFTv2, an input-centric reformulation that instead uses matrix-vector multiplications (i.e., matrix-free computation), reducing the computational cost to quadratic.<n>These modifications allow OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage without compromising performance.
  arXiv  Detail & Related papers  (2025-06-24T17:59:49Z)
• Design and Implementation of an FPGA-Based Tiled Matrix Multiplication Accelerator for Transformer Self-Attention on the Xilinx KV260 SoM [0.0]
  Transformer-based large language models rely heavily on matrix multiplications for attention and feed-forward layers. We introduce a highly optimized tiled matrix multiplication accelerator on a resource-constrained Xilinx KV260 FPGA. Our design exploits persistent on-chip storage, a robust two-level tiling strategy for maximal data reuse, and a systolic-like unrolled compute engine.
  arXiv  Detail & Related papers  (2025-03-20T22:15:42Z)
• LLM Inference Acceleration via Efficient Operation Fusion [1.350507740574158]
  Transformer-based Large Language Models (LLMs) contain hundreds of billions of parameters and require dedicated hardware resources for training and inference. One of the key challenges inherent to the Transformer architecture is the requirement to support numerous non-linear transformations. We propose an extremely efficient technique that can completely hide the overhead caused by such collective operations.
  arXiv  Detail & Related papers  (2025-02-24T23:42:37Z)
• Machine learning-driven conservative-to-primitive conversion in hybrid piecewise polytropic and tabulated equations of state [0.2999888908665658]
  We present a novel machine learning (ML) method to accelerate conservative-to-primitive inversion in hydrodynamics simulations. We employ feedforward neural networks (NNC2PS and NNC2PL), trained in PyTorch, and optimized for GPU inference using NVIDIART. The mixed-precisionRT engine for NNC2PS inference speeds approximately 400 times faster than a traditional single-threaded implementation for a dataset size of 1,000,000 points.
  arXiv  Detail & Related papers  (2024-12-10T19:00:01Z)
• Optimizing the Deployment of Tiny Transformers on Low-Power MCUs [12.905978154498499]
  This work aims to enable and optimize the flexible, multi-platform deployment of encoder Tiny Transformers on commercial MCUs. Our framework provides an optimized library of kernels to maximize data reuse and avoid data marshaling operations into the crucial attention block. We show that our MHSA depth-first tiling scheme reduces the memory peak by up to 6.19x, while the fused-weight attention can reduce the runtime by 1.53x, and number of parameters by 25%.
  arXiv  Detail & Related papers  (2024-04-03T14:14:08Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• Gated Linear Attention Transformers with Hardware-Efficient Training [60.670102007737476]
  This work describes a hardware-efficient algorithm for linear attention that trades off memory movement against parallelizability. We then generalize this algorithm to a more expressive variant of linear attention with data-dependent gates. When used as a replacement for the standard attention layer in Transformers, the resulting gated linear attention Transformer is found to perform competitively.
  arXiv  Detail & Related papers  (2023-12-11T18:51:59Z)
• TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing [52.64837396100988]
  MEGA is a recent transformer-based architecture, which utilizes a linear recurrent operator whose parallel computation, based on the FFT, scales as $O(LlogL)$, with $L$ being the sequence length. We build upon their approach by replacing the linear recurrence with a special temporal convolutional network which permits larger receptive field size with shallower networks, and reduces the computational complexity to $O(L)$. We evaluate TCNCA on EnWik8 language modeling, long-range-arena (LRA) sequence classification, as well as a synthetic reasoning benchmark associative recall.
  arXiv  Detail & Related papers  (2023-12-09T16:12:25Z)
• ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation [2.7488316163114823]
  This paper proposes a novel approach to an energy-efficient acceleration of frequency-domain neural networks by utilizing analog-domain frequency-based tensor transformations. Our approach achieves more compact cells by eliminating the need for trainable parameters in the transformation matrix. On a 16$times$16 crossbars, for 8-bit input processing, the proposed approach achieves the energy efficiency of 1602 tera operations per second per Watt.
  arXiv  Detail & Related papers  (2023-09-04T19:19:39Z)
• Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers [15.37318446043671]
  This paper introduces a novel reduced precision optimization technique for On-Device Learning (ODL) primitives on MCU-class devices. Our approach results more than two orders of magnitude faster than existing ODL software frameworks for single-core MCUs.
  arXiv  Detail & Related papers  (2023-05-30T16:14:16Z)
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale [80.86029795281922]
  We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers. A 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation.
  arXiv  Detail & Related papers  (2022-08-15T17:08:50Z)
• An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers [11.811907838840712]
  We propose an algorithm-hardware co-optimized framework to flexibly and efficiently accelerate Transformers by utilizing general N:M sparsity patterns. We present a flexible and efficient hardware architecture, namely STA, to achieve significant speedup when deploying N:M sparse Transformers. Experimental results show that compared to other methods, N:M sparse Transformers, generated using IDP, achieves an average of 6.7% improvement on accuracy with high training efficiency.
  arXiv  Detail & Related papers  (2022-08-12T04:51:49Z)
• Reducing Activation Recomputation in Large Transformer Models [17.810669621463962]
  We show how to significantly accelerate training of large transformer models by reducing activation recomputation. We present two novel yet very simple techniques: sequence parallelism and selective activation recomputation. Our method reduces activation memory by 5x, while reducing execution time overhead from activation recomputation by over 90%.
  arXiv  Detail & Related papers  (2022-05-10T22:40:17Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)
• Deriving Differential Target Propagation from Iterating Approximate Inverses [91.3755431537592]
  We show that a particular form of target propagation, relying on learned inverses of each layer, which is differential, gives rise to an update rule which corresponds to an approximate Gauss-Newton gradient-based optimization. We consider several iterative calculations based on local auto-encoders at each layer in order to achieve more precise inversions for more accurate target propagation.
  arXiv  Detail & Related papers  (2020-07-29T22:34:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.