Length-Aware Rotary Position Embedding for Text-Speech Alignment

• URL: http://arxiv.org/abs/2509.11084v1
• Date: Sun, 14 Sep 2025 04:25:13 GMT
• Title: Length-Aware Rotary Position Embedding for Text-Speech Alignment
• Authors: Hyeongju Kim, Juheon Lee, Jinhyeok Yang, Jacob Morton,
• Abstract summary: We introduce length-aware RoPE (LARoPE), a simple yet effective extension of RoPE that improves text-speech alignment.<n> Experimental results show that LARoPE consistently outperforms RoPE, offering faster loss convergence, more accurate text-speech alignment, and higher overall TTS quality.
• Score: 8.321525172143609
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Many recent text-to-speech (TTS) systems are built on transformer architectures and employ cross-attention mechanisms for text-speech alignment. Within these systems, rotary position embedding (RoPE) is commonly used to encode positional information in text and speech representations. In this work, we introduce length-aware RoPE (LARoPE), a simple yet effective extension of RoPE that improves text-speech alignment. Unlike RoPE, which relies on absolute indices, LARoPE computes relative distances between query and key positions using length-normalized indices. Experimental results show that LARoPE consistently outperforms RoPE, offering faster loss convergence, more accurate text-speech alignment, and higher overall TTS quality. Furthermore, LARoPE demonstrates greater resilience to variations in utterance duration and maintains stable performance in extended speech generation up to 30 seconds, whereas RoPE suffers from notable degradation. Notably, our method achieves a state-of-the-art word error rate on a standard zero-shot TTS benchmark.

Related papers

• CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs [18.897130541385646]
  Rotary Positional Embedding (RoPE) is a key component of context scaling in Large Language Models (LLMs)<n>In this work, we unify these seemingly distinct objectives through a minimalist intervention, namely soft clipping lowfrequency components of RoPE.<n>CoPE not only eliminates OOD outliers and refines semantic signals, but also prevents spectral leakage caused by hard clipping.
  arXiv  Detail & Related papers  (2026-02-05T03:31:14Z)
• DoPE: Denoising Rotary Position Embedding [60.779039511252584]
  Rotary Position Embedding (RoPE) in Transformer models has inherent limits that weaken length.<n>We reinterpret the attention map with positional encoding as a noisy feature map, and propose Denoising Positional extrapolation page (DoPE)<n>DoPE is a training-free method based on truncated matrix entropy to detect outlier frequency bands in the feature map.
  arXiv  Detail & Related papers  (2025-11-12T09:32:35Z)
• Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings [29.421443764865003]
  We present an analysis indicating that what and where are entangled in the popular RoPE rotary position embedding.<n>We propose an improvement to RoPE, which we call Polar Coordinate Position Embeddings or PoPE, that eliminates the what-where confound.
  arXiv  Detail & Related papers  (2025-09-05T14:22:27Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• VRoPE: Rotary Position Embedding for Video Large Language Models [13.495442349395287]
  Position Embedding (RoPE) has shown strong performance in text-based Large Language Models (LLMs)<n>Video adaptations, such as RoPE-3D, attempt to encode spatial and temporal dimensions separately but suffer from two major limitations.<n>We propose Position Rotary Embedding (VRoPE), a novel positional encoding method tailored for Video-LLMs.
  arXiv  Detail & Related papers  (2025-02-17T10:53:57Z)
• VideoRoPE: What Makes for Good Video Rotary Position Embedding? [109.88966080843608]
  VideoRoPE consistently surpasses previous RoPE variants, across diverse downstream tasks such as long video retrieval, video understanding, and video hallucination.<n>VideoRoPE features textlow-frequency temporal allocation to mitigate periodic oscillations, a textitdiagonal layout to maintain spatial symmetry, and textadjustable temporal spacing to decouple temporal and spatial indexing.
  arXiv  Detail & Related papers  (2025-02-07T18:56:04Z)
• Benchmarking Rotary Position Embeddings for Automatic Speech Recognition [17.360059094663182]
  Relative Position (RelPos) embeddings are widely used in Automatic Speech Recognition (ASR)<n>In contrast, Rotary Positional Embedding (RoPE) rotates each input vector based on its absolute position, taking linear time to sequence length.<n>This work evaluates RoPE across diverse ASR tasks with training data ranging from 100 to 50,000 hours.
  arXiv  Detail & Related papers  (2025-01-10T15:30:46Z)
• Rotary Position Embedding for Vision Transformer [44.27871591624888]
  This study provides a comprehensive analysis of Rotary Position Embedding (RoPE) when applied to Vision Transformer (ViT) RoPE demonstrates impressive extrapolation performance, i.e., maintaining precision while increasing image resolution at inference. It eventually leads to performance improvement for ImageNet-1k, COCO detection, and ADE-20k segmentation.
  arXiv  Detail & Related papers  (2024-03-20T04:47:13Z)
• Scaling Laws of RoPE-based Extrapolation [103.33995311915864]
  We propose textbftextitScaling Laws of RoPE-based Extrapolation to describe the relationship between the extrapolation performance and base value. We achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.
  arXiv  Detail & Related papers  (2023-10-08T15:50:36Z)
• Conformer-based End-to-end Speech Recognition With Rotary Position Embedding [11.428057887454008]
  We introduce rotary position embedding (RoPE) in the convolution-augmented transformer (conformer) RoPE encodes absolute positional information into the input sequence by a rotation matrix, and then naturally incorporates explicit relative position information into a self-attention module. Our model achieves a relative word error rate reduction of 8.70% and 7.27% over the conformer on test-clean and test-other sets of the LibriSpeech corpus respectively.
  arXiv  Detail & Related papers  (2021-07-13T08:07:22Z)
• Dynamic Acoustic Unit Augmentation With BPE-Dropout for Low-Resource End-to-End Speech Recognition [62.94773371761236]
  We consider building an effective end-to-end ASR system in low-resource setups with a high OOV rate. We propose a method of dynamic acoustic unit augmentation based on the BPE-dropout technique. Our monolingual Turkish Conformer established a competitive result with 22.2% character error rate (CER) and 38.9% word error rate (WER)
  arXiv  Detail & Related papers  (2021-03-12T10:10:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.