Reducing Tokenization Premiums for Low-Resource Languages

• URL: http://arxiv.org/abs/2601.13328v1
• Date: Mon, 19 Jan 2026 19:08:58 GMT
• Title: Reducing Tokenization Premiums for Low-Resource Languages
• Abstract summary: Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs.<n>This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages.<n>We propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens.
• Score: 5.02470728447561
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs, meaning that it generally requires several times as many tokens to encode a sentence in a low-resource language than to encode the analogous sentence in English. This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages. In this paper we analyze the tokenizers of ten popular LMs to better understand their designs and per-language tokenization premiums. We also propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens. We apply this methodology to 12 low-resource languages, demonstrating that the original and compressed inputs often have similar last hidden states when run through the Llama 3.2 1B model.

Related papers

• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages [0.0]
  Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.<n>We quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus.<n>We show that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents.
  arXiv  Detail & Related papers  (2026-07-27T11:20:29Z)
• In-Place Tokenizer Expansion for Pre-trained LLMs [47.56305202796458]
  We present an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design.<n>A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality.<n>The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4times$ and $2.6times$ fewer tokens than the source.
  arXiv  Detail & Related papers  (2026-07-16T17:32:38Z)
• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• Explaining and Mitigating Crosslingual Tokenizer Inequities [18.823984182115712]
  Having high token premiums leads to less throughput during training and increases costs at inference.<n>We train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages.<n>We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do.
  arXiv  Detail & Related papers  (2025-10-24T17:36:03Z)
• The Token Tax: Systematic Bias in Multilingual Tokenization [0.8820808252713569]
  Tokenization inefficiency imposes structural disadvantages on morphologically complex, low-resource languages.<n>We show that fertility (tokens/word) reliably predicts accuracy.<n>We also find that reasoning models (DeepSeek, o1) consistently outperform non-reasoning peers.
  arXiv  Detail & Related papers  (2025-09-05T20:20:51Z)
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [53.22544362024936]
  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.<n>Standard algorithms for learning tokenizers rely on frequency-based objectives.<n>We introduce Parity-aware Byte Pair.<n>We find empirically that Parity-aware BPE leads to more equitable token counts across languages.
  arXiv  Detail & Related papers  (2025-08-06T18:14:43Z)
• Sampling from Your Language Model One Byte at a Time [82.71473348639489]
  Tokenization can introduce distortion into the model's generations, known as the Prompt Boundary Problem (PBP)<n>We present an inference-time method to convert any autore LM with a BPE tokenizer into a character-level or byte-level LM.<n>Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers.
  arXiv  Detail & Related papers  (2025-06-17T02:37:04Z)
• Comparative analysis of subword tokenization approaches for Indian languages [5.012314384895538]
  Tokenization is the act of breaking down text into smaller parts, or tokens, that are easier for machines to process.<n>Subword tokenization enhances this process by breaking down words into smaller subword units.<n>It is useful in capturing the intricate structure of words in Indian languages (ILs), such as prefixes, suffixes, and other morphological variations.<n>This paper examines how different subword tokenization techniques, such as SentencePiece, Byte Pair, and WordPiece Tokenization, affect ILs.
  arXiv  Detail & Related papers  (2025-05-22T16:24:37Z)
• Beyond Literal Token Overlap: Token Alignability for Multilinguality [53.680462160878925]
  We propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation.<n>In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low.<n>We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2025-02-10T13:50:12Z)
• From Language Models over Tokens to Language Models over Characters [54.123846188068384]
  Modern language models are internally -- and mathematically -- distributions over $ittoken$ strings rather than $itcharacter$ strings.<n>This paper presents algorithms for converting token-level language models to character-level ones.
  arXiv  Detail & Related papers  (2024-12-04T21:19:20Z)
• Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation [19.624330093598996]
  Training monolingual language models for low and mid-resource languages is made challenging by limited and often inadequate pretraining data. By generalizing over a word translation dictionary encompassing both the source and target languages, we map tokens from the target tokenizer to semantically similar tokens from the source language tokenizer. We conduct experiments to convert high-resource models to mid- and low-resource languages, namely Dutch and Frisian.
  arXiv  Detail & Related papers  (2023-10-05T11:45:29Z)
• Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models [68.29126169579132]
  API vendors charge their users based on usage, more specifically on the number of tokens'' processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. We conduct a systematic analysis of the cost and utility of OpenAI's language model API on multilingual benchmarks in 22 typologically diverse languages.
  arXiv  Detail & Related papers  (2023-05-23T05:46:45Z)
• Crowdsourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The Case of Fon Language [0.015863809575305417]
  We introduce Word-Expressions-Based (WEB) tokenization, a human-involved super-words tokenization strategy to create a better representative vocabulary for training. We compare our tokenization strategy to others on the Fon-French and French-Fon translation tasks.
  arXiv  Detail & Related papers  (2021-03-14T22:12:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.