The Token Tax: Systematic Bias in Multilingual Tokenization

• URL: http://arxiv.org/abs/2509.05486v1
• Date: Fri, 05 Sep 2025 20:20:51 GMT
• Title: The Token Tax: Systematic Bias in Multilingual Tokenization
• Abstract summary: Tokenization inefficiency imposes structural disadvantages on morphologically complex, low-resource languages.<n>We show that fertility (tokens/word) reliably predicts accuracy.<n>We also find that reasoning models (DeepSeek, o1) consistently outperform non-reasoning peers.
• Score: 0.8820808252713569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tokenization inefficiency imposes structural disadvantages on morphologically complex, low-resource languages, inflating compute resources and depressing accuracy. We evaluate 10 large language models (LLMs) on AfriMMLU (9,000 MCQA items; 5 subjects; 16 African languages) and show that fertility (tokens/word) reliably predicts accuracy. Higher fertility consistently predicts lower accuracy across all models and subjects. We further find that reasoning models (DeepSeek, o1) consistently outperform non-reasoning peers across high and low resource languages in the AfriMMLU dataset, narrowing accuracy gaps observed in prior generations. Finally, translating token inflation to economics, a doubling in tokens results in quadrupled training cost and time, underscoring the token tax faced by many languages. These results motivate morphologically aware tokenization, fair pricing, and multilingual benchmarks for equitable natural language processing (NLP).

Related papers

• Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages [0.3441021278275805]
  Pretraining LLMs on artificial languages ("pre-pretraining") is a technique that could reportedly increase token efficiency by 33%.<n>We validate this prior result for English on a larger set of natural languages across four language families.<n>We also relate the observed gains (or losses) in token efficiency to quantified linguistic properties of the languages.
  arXiv  Detail & Related papers  (2026-08-09T16:26:29Z)
• AdaMame: A Training Recipe for Adaptive Multilingual Reasoning [32.31269959716733]
  Large Reasoning Models (LRMs) often fail to reason in the language of the query.<n>AdaMame is a two-stage training recipe for multilingual mathematical reasoning.
  arXiv  Detail & Related papers  (2026-06-13T03:22:35Z)
• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models [30.7730386396743]
  Byte-level Byte-Pair tokenizers structurally favor high-resource languages and Latin scripts.<n>For speakers of underrepresented languages, this bias inflates inference costs and widens cross-lingual capability gaps.<n>We present the first systematic comparison of equitable tokenizers on a benchmark spanning 11 Southeast Asian languages.
  arXiv  Detail & Related papers  (2026-06-13T01:10:42Z)
• Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation [49.2073409243885]
  Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency.<n>We conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages.<n>We identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages.
  arXiv  Detail & Related papers  (2026-01-01T08:53:49Z)
• Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiency [6.943451388015595]
  Tokenization disparities pose a significant barrier to achieving equitable access to artificial intelligence.<n>This study conducts a large-scale cross-linguistic evaluation of tokenization efficiency in over 200 languages.
  arXiv  Detail & Related papers  (2025-10-14T11:14:38Z)
• Low-Resource English-Tigrinya MT: Leveraging Multilingual Models, Custom Tokenizers, and Clean Evaluation Benchmarks [6.177998679139308]
  Despite advances in Neural Machine Translation (NMT), low-resource languages like Tigrinya remain underserved.<n>This paper investigates transfer learning techniques using multilingual pretrained models to enhance translation quality for morphologically rich, low-resource languages.
  arXiv  Detail & Related papers  (2025-09-24T15:02:57Z)
• Evaluating LLMs' Multilingual Capabilities for Bengali: Benchmark Creation and Performance Analysis [0.0]
  Bengali is an underrepresented language in NLP research.<n>We systematically investigate the challenges that hinder Bengali NLP performance.<n>Our findings reveal consistent performance gaps for Bengali compared to English.
  arXiv  Detail & Related papers  (2025-07-31T05:16:43Z)
• Information Locality as an Inductive Bias for Neural Language Models [52.92279412466086]
  We show that $m$local entropy are difficult for Transformer and LSTM LMs to learn languages.<n>These results suggest that neurals are highly sensitive to the statistical structure of a language.
  arXiv  Detail & Related papers  (2025-06-05T15:21:05Z)
• Towards Typologically Aware Rescoring to Mitigate Unfaithfulness in Lower-Resource Languages [9.426642998924724]
  Multilingual large language models generate non-faithful output in resource-constrained languages.<n>To mitigate unfaithfulness in such settings, we propose using computationally light auxiliary models to rescore the outputs of larger architectures.<n>We show that monolingual 4-layer BERT models pretrained from scratch on less than 700 MB of data without fine-tuning are able to identify faithful summaries with a mean accuracy of 88.33%.
  arXiv  Detail & Related papers  (2025-02-24T21:22:19Z)
• How does a Language-Specific Tokenizer affect LLMs? [0.36248657646376703]
  The necessity of language-specific tokenizers intuitively appears crucial for effective natural language processing.<n>This study explores how language-specific tokenizers influence the behavior of Large Language Models predominantly trained with English text data.
  arXiv  Detail & Related papers  (2025-02-18T05:54:56Z)
• Towards Faster k-Nearest-Neighbor Machine Translation [51.866464707284635]
  k-nearest-neighbor machine translation approaches suffer from heavy retrieve overhead on the entire datastore when decoding each token.<n>We propose a simple yet effective multi-layer perceptron (MLP) network to predict whether a token should be translated jointly by the neural machine translation model and probabilities produced by the kNN.<n>Our method significantly reduces the overhead of kNN retrievals by up to 53% at the expense of a slight decline in translation quality.
  arXiv  Detail & Related papers  (2023-12-12T16:41:29Z)
• How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training? [86.48323488619629]
  We analyze how translation performance changes as the data ratios among languages vary in the tokenizer training corpus. We find that while relatively better performance is often observed when languages are more equally sampled, the downstream performance is more robust to language imbalance than we usually expected.
  arXiv  Detail & Related papers  (2022-04-29T17:50:36Z)
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension [51.953428342923885]
  We develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning mechanism is developed to learn the fine difference between the accurate answer and other candidates.
  arXiv  Detail & Related papers  (2021-12-09T07:31:15Z)
• Emergent Communication Pretraining for Few-Shot Machine Translation [66.48990742411033]
  We pretrain neural networks via emergent communication from referential games. Our key assumption is that grounding communication on images---as a crude approximation of real-world environments---inductively biases the model towards learning natural languages.
  arXiv  Detail & Related papers  (2020-11-02T10:57:53Z)
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation [81.7786241489002]
  Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics. We propose random online backtranslation to enforce the translation of unseen training language pairs.
  arXiv  Detail & Related papers  (2020-04-24T17:21:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.