BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition

• URL: http://arxiv.org/abs/2602.01717v1
• Date: Mon, 02 Feb 2026 06:56:27 GMT
• Title: BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition
• Abstract summary: Byte-level BPE (BBPE) using Longer-8 is widely adopted for its language-agnostic design and full coverage.<n>We propose BBPE16, a Unicode-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit.
• Score: 18.054653205555862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual automatic speech recognition (ASR) requires tokenization that efficiently covers many writing systems. Byte-level BPE (BBPE) using UTF-8 is widely adopted for its language-agnostic design and full Unicode coverage, but its variable-length encoding inflates token sequences for non-Latin scripts, such as Chinese, Japanese, and Korean (CJK). Longer sequences increase computational load and memory use. We propose BBPE16, a UTF-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit. BBPE16 preserves BBPE's language-agnostic properties while substantially improving cross-lingual token sharing. Across monolingual, bilingual, and trilingual ASR, and in a multilingual continual-learning setup, BBPE16 attains comparable or better accuracy; for Chinese, it reduces token counts by up to 10.4% and lowers decoding iterations by up to 10.3%. These reductions speed up fine-tuning and inference and decrease memory usage, making BBPE16 a practical tokenization choice for multilingual ASR.

Related papers

• Token-Native Storage: Read and Write in your Agent's Language [0.0]
  We argue for token-native storage of text as the model's own byte-pair-encoding (BPE) token IDs.<n>BPE numbers tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer (streamvbyte) recover most of the entropy coder's ratio.<n>Because a model reads token IDs, not text, a token-native store hands over the IDs directly instead of retokenizing on every read.
  arXiv  Detail & Related papers  (2026-08-03T15:20:36Z)
• Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet [15.63204319501313]
  We propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers.<n> IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution.<n>We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality.
  arXiv  Detail & Related papers  (2026-06-18T23:50:54Z)
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [53.22544362024936]
  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.<n>Standard algorithms for learning tokenizers rely on frequency-based objectives.<n>We introduce Parity-aware Byte Pair.<n>We find empirically that Parity-aware BPE leads to more equitable token counts across languages.
  arXiv  Detail & Related papers  (2025-08-06T18:14:43Z)
• BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization [0.0]
  SCRIPT (Script Category Representation in PreTokenization) is a novel encoding scheme that bypasses byte-8 conversion by using initial tokens based on Unicode script and category properties.<n>Our experiments demonstrate that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties for non-Latin-script languages.
  arXiv  Detail & Related papers  (2025-05-30T15:12:41Z)
• MorphTok: Morphologically Grounded Tokenization for Indian Languages [18.594241501479747]
  Tokenization is a crucial step in NLP, especially with the rise of large language models (LLMs)<n>We propose morphology-aware segmentation as a pre-tokenization step before applying the classical Byte-pair.<n>To handle the dependent vowels common in syllable-based writing systems, we propose Constrained BPE (CBPE)<n>CBPE handles dependent vowels to form a cohesive unit with other characters instead of occurring as a single unit.
  arXiv  Detail & Related papers  (2025-04-14T15:44:45Z)
• Beyond Literal Token Overlap: Token Alignability for Multilinguality [53.680462160878925]
  We propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation.<n>In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low.<n>We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2025-02-10T13:50:12Z)
• Optimizing Byte-level Representation for End-to-end ASR [4.247168417243709]
  We propose a novel approach to optimize a byte-level representation for end-to-end automatic speech recognition (ASR) Byte-level representation is often used by large scale ASR systems when the character set of the supported languages is large. In an English/Mandarin dictation task, we show that a bilingual ASR model built with this approach can outperform by 5% relative in error rate.
  arXiv  Detail & Related papers  (2024-06-14T02:58:19Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods typically align vision encoders with Multimodal Large Language Models (MLLMs) via supervised fine-tuning (SFT)<n>We propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level.<n>We introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling [70.34758460372629]
  We introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. MYTE produces shorter encodings for all 99 analyzed languages. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
  arXiv  Detail & Related papers  (2024-03-15T21:21:11Z)
• MRN: Multiplexed Routing Network for Incremental Multilingual Text Recognition [56.408324994409405]
  Multiplexed routing network (MRN) trains a recognizer for each language that is currently seen. MRN effectively reduces the reliance on older data and better fights against catastrophic forgetting. It outperforms existing general-purpose IL methods by large margins.
  arXiv  Detail & Related papers  (2023-05-24T06:03:34Z)
• Scaling Up Deliberation for Multilingual ASR [36.860327600638705]
  We investigate second-pass deliberation for multilingual speech recognition. Our proposed deliberation is multilingual, i.e., the text encoder encodes hypothesis text from multiple languages, and the decoder attends to multilingual text and audio. We show that deliberation improves the average WER on 9 languages by 4% relative compared to the single-pass model.
  arXiv  Detail & Related papers  (2022-10-11T21:07:00Z)
• LAE: Language-Aware Encoder for Monolingual and Multilingual ASR [87.74794847245536]
  A novel language-aware encoder (LAE) architecture is proposed to handle both situations by disentangling language-specific information. Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level.
  arXiv  Detail & Related papers  (2022-06-05T04:03:12Z)
• Efficient Weight factorization for Multilingual Speech Recognition [67.00151881207792]
  End-to-end multilingual speech recognition involves using a single model training on a compositional speech corpus including many languages. Due to the fact that each language in the training data has different characteristics, the shared network may struggle to optimize for all various languages simultaneously. We propose a novel multilingual architecture that targets the core operation in neural networks: linear transformation functions.
  arXiv  Detail & Related papers  (2021-05-07T00:12:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.