Explaining and Mitigating Crosslingual Tokenizer Inequities

• URL: http://arxiv.org/abs/2510.21909v1
• Date: Fri, 24 Oct 2025 17:36:03 GMT
• Title: Explaining and Mitigating Crosslingual Tokenizer Inequities
• Abstract summary: Having high token premiums leads to less throughput during training and increases costs at inference.<n>We train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages.<n>We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do.
• Score: 18.823984182115712
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less throughput during training and increases costs at inference. In this paper, we show that even after controlling for dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of token premiums across languages. To understand the cross-linguistic differences that cause these token premiums, we train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages, manipulating tokenization algorithm, vocabulary size, and dataset size. We measure token premiums and test for a relationship between factors such as data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of language-specific features such as writing system and word length. We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do. While simply increasing vocabulary size does not lead to reduced token premium effects, we can determine an ``optimal'' vocabulary size for each language to achieve significantly reduced token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce token premium effects and improve compression overall. Thus, intervening on the vocabulary size or the pre-tokenizer significantly reduces crosslingual token premium effects.

Related papers

• TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment [46.819665083185136]
  Inefficient tokenization results in long token-ID sequences and will slow down the training and inference of Large Language Models.<n>We introduce a method named TokAlign++ to improve vocabulary adaptation performance by learning better token alignment lexicon.<n> Experimental results on 15 languages show that our method boosts the multilingual text compression rates and preserves most of the multilingual ability of vanilla models.
  arXiv  Detail & Related papers  (2026-05-13T12:23:24Z)
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• Causal Estimation of Tokenisation Bias [58.20086589761273]
  We quantify the effect of including or not a subword in a tokeniser's vocabulary on the probability a trained model assigns to the corresponding characters.<n>We find that tokenisation consistently affects models' outputs across scales, vocabularies, and tokenisers.<n> Notably, a subword's presence in a small model's vocabulary may increase its characters' probability by up to 17 times.
  arXiv  Detail & Related papers  (2025-06-03T17:59:47Z)
• Comparative analysis of subword tokenization approaches for Indian languages [5.012314384895538]
  Tokenization is the act of breaking down text into smaller parts, or tokens, that are easier for machines to process.<n>Subword tokenization enhances this process by breaking down words into smaller subword units.<n>It is useful in capturing the intricate structure of words in Indian languages (ILs), such as prefixes, suffixes, and other morphological variations.<n>This paper examines how different subword tokenization techniques, such as SentencePiece, Byte Pair, and WordPiece Tokenization, affect ILs.
  arXiv  Detail & Related papers  (2025-05-22T16:24:37Z)
• SuperBPE: Space Travel for Language Models [103.09169510391972]
  We introduce a "superword" tokenizer, SuperBPE, which incorporates a simple pretokenization curriculum into the byte-pair encoding (BPE) algorithm.<n>SuperBPE encodes a fixed piece of text with up to 33% fewer tokens than BPE on average.<n>Our model trained with SuperBPE achieves an average +4.0% absolute improvement over the BPE baseline across 30 downstream tasks.
  arXiv  Detail & Related papers  (2025-03-17T17:53:23Z)
• How Much is Enough? The Diminishing Returns of Tokenization Training Data [4.300681074103876]
  We train BPE, UnigramLM, and WordPiece tokenizers across various vocabulary sizes using English training data ranging from 1GB to 900GB.<n>Our findings reveal diminishing returns as training data size increases beyond roughly 150GB.<n>For Russian text, we observe diminishing returns after training a tokenizer from 200GB of data, which is approximately 33% more than when training on English.
  arXiv  Detail & Related papers  (2025-02-27T17:01:23Z)
• Beyond Literal Token Overlap: Token Alignability for Multilinguality [53.680462160878925]
  We propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation.<n>In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low.<n>We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2025-02-10T13:50:12Z)
• Tokenization Is More Than Compression [14.939912120571728]
  Existing tokenization approaches like Byte-Pair. (BPE) originate from the field of data compression. We introduce PathPiece, a new tokenizer that segments a document's text into the minimum number of tokens for a given vocabulary.
  arXiv  Detail & Related papers  (2024-02-28T14:52:15Z)
• Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers [48.036317742487796]
  We propose a new approach to tokenization for lexical matching retrieval algorithms. We use the WordPiece tokenizer, which can be built automatically from unsupervised data. Results show that the mBERT tokenizer provides strong relevance signals for retrieval "out of the box", outperforming whitespace tokenization on most languages.
  arXiv  Detail & Related papers  (2022-10-11T14:32:46Z)
• Impact of Tokenization on Language Models: An Analysis for Turkish [2.4660652494309936]
  We train tokenizers and pretrain medium-sized language models using RoBERTa pretraining procedure on the Turkish split of the OSCAR corpus. Our experiments, supported by statistical tests, reveal that Morphological-level tokenizer has challenging performance with de facto tokenizers. We find that increasing the vocabulary size improves the performance of Morphological and Word-level tokenizers more than that of de facto tokenizers.
  arXiv  Detail & Related papers  (2022-04-19T12:01:46Z)
• More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models [71.42030830910227]
  We propose a new metric for measuring the clustering quality in settings where the models differ. We show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.
  arXiv  Detail & Related papers  (2021-08-24T14:08:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.