Efficient Matrix Implementation for Rotary Position Embedding

• URL: http://arxiv.org/abs/2604.09742v1
• Date: Fri, 10 Apr 2026 00:17:47 GMT
• Title: Efficient Matrix Implementation for Rotary Position Embedding
• Abstract summary: Rotary Position Embedding (RoPE) has become a core component of modern Transformer architectures across language, vision, and 3D domains.<n>We propose RoME, a mathematically equivalent yet computationally efficient reformulation of RoPE that replaces vector operations with unified matrix transformations.<n> experiments show that RoME delivers substantial acceleration at both the operator and full-model levels.
• Score: 50.45885099010382
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rotary Position Embedding (RoPE) has become a core component of modern Transformer architectures across language, vision, and 3D domains. However, existing implementations rely on vector-level split and merge operations that introduce non-negligible computational overhead, often overlooked in attention optimization. The problem is further amplified in multi-dimensional settings (e.g., 2D and 3D RoPE), where additional vector operations and uneven feature partitions degrade hardware utilization. To overcome these limitations, we propose RoME (Rotary Matrix position Embedding), a mathematically equivalent yet computationally efficient reformulation of RoPE that replaces vector operations with unified matrix transformations. RoME eliminates dimension-specific operations, simplifies implementation, and enables fused parallel execution across Cube and Vector units on modern NPUs. Experiments show that RoME delivers substantial acceleration at both the operator and full-model levels. The implementation is available at https://gitcode.com/cann/ops-transformer/blob/master/experimental/posembedding/rope_matrix/README.md .

Related papers

• RSR-core: A High-Performance Engine for Low-Bit Matrix-Vector Multiplication [11.676571773958145]
  Matrix-vector multiplication is a fundamental building block in neural networks, vector databases, and large language models, particularly during inference.<n>Recent work has explored low-bit quantization of model weights, where matrices are represented using binary (1-bit) or ternary (1.58-bit) values while activation is kept in higher precision.<n>In parallel, algorithms such as Redundant Segment Reduction provide theoretical guarantees for accelerating low-bit matrix-vector multiplication.
  arXiv  Detail & Related papers  (2026-03-29T00:55:14Z)
• NeuMatC: A General Neural Framework for Fast Parametric Matrix Operation [75.91285900600549]
  We propose textbftextitNeural Matrix Computation Framework (NeuMatC), which elegantly tackles general parametric matrix operation tasks.<n>NeuMatC unsupervisedly learns a low-rank and continuous mapping from parameters to their corresponding matrix operation results.<n> Experimental results on both synthetic and real-world datasets demonstrate the promising performance of NeuMatC.
  arXiv  Detail & Related papers  (2025-11-28T07:21:17Z)
• TriADA: Massively Parallel Trilinear Matrix-by-Tensor Multiply-Add Algorithm and Device Architecture for the Acceleration of 3D Discrete Transformations [0.0]
  Multilinear transformations are key in high-performance computing ( HPC) and artificial intelligence (AI) workloads.<n> scaling computation by enlarging the number of parallel processing units substantially increases energy consumption.<n>TriADA is capable of performing a variety of trilinear transformations with hypercubic arithmetic complexity in a linear number of time-steps.
  arXiv  Detail & Related papers  (2025-06-28T08:42:01Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• LLM Inference Acceleration via Efficient Operation Fusion [1.350507740574158]
  Transformer-based Large Language Models (LLMs) contain hundreds of billions of parameters and require dedicated hardware resources for training and inference.<n>One of the key challenges inherent to the Transformer architecture is the requirement to support numerous non-linear transformations.<n>We propose an extremely efficient technique that can completely hide the overhead caused by such collective operations.
  arXiv  Detail & Related papers  (2025-02-24T23:42:37Z)
• LieRE: Lie Rotational Positional Encodings [5.32707456872718]
  Transformer architectures rely on position encodings to model the structure of input data.<n>We introduce Lie Relative algebras (LieRE) to increase the representational capacity of positional encodings in transformers.<n>We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions.
  arXiv  Detail & Related papers  (2024-06-14T17:41:55Z)
• DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets [95.84755169585492]
  We present Dynamic Sparse Voxel Transformer (DSVT), a single-stride window-based voxel Transformer backbone for outdoor 3D perception. Our model achieves state-of-the-art performance with a broad range of 3D perception tasks.
  arXiv  Detail & Related papers  (2023-01-15T09:31:58Z)
• TransCMD: Cross-Modal Decoder Equipped with Transformer for RGB-D Salient Object Detection [86.94578023985677]
  In this work, we rethink this task from the perspective of global information alignment and transformation. Specifically, the proposed method (TransCMD) cascades several cross-modal integration units to construct a top-down transformer-based information propagation path. Experimental results on seven RGB-D SOD benchmark datasets demonstrate that a simple two-stream encoder-decoder framework can surpass the state-of-the-art purely CNN-based methods.
  arXiv  Detail & Related papers  (2021-12-04T15:45:34Z)
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding [63.539333383965726]
  We propose a novel way to accelerate attention calculation for Transformers with relative positional encoding (RPE) Based upon the observation that relative positional encoding forms a Toeplitz matrix, we mathematically show that kernelized attention with RPE can be calculated efficiently using Fast Fourier Transform (FFT)
  arXiv  Detail & Related papers  (2021-06-23T17:51:26Z)
• Oriented Object Detection with Transformer [51.634913687632604]
  We implement Oriented Object DEtection with TRansformer ($bf O2DETR$) based on an end-to-end network. We design a simple but highly efficient encoder for Transformer by replacing the attention mechanism with depthwise separable convolution. Our $rm O2DETR$ can be another new benchmark in the field of oriented object detection, which achieves up to 3.85 mAP improvement over Faster R-CNN and RetinaNet.
  arXiv  Detail & Related papers  (2021-06-06T14:57:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.