BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

• URL: http://arxiv.org/abs/2605.29379v1
• Date: Thu, 28 May 2026 05:29:12 GMT
• Title: BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base
• Abstract summary: BrahmicTokenizer-131K is a 131,072-vocabulary byte-level BPE tokenizer.<n>It closes the Brahmic compression gap at the 131K-vocabulary class.<n>It preserves the English, EU-language, and code compression of OpenAI's o200k_base.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present BrahmicTokenizer-131K, a 131,072-vocabulary byte-level BPE tokenizer that closes the Brahmic compression gap at the 131K-vocabulary class while preserving the English, EU-language, and code compression of OpenAI's o200k_base. We construct it through a two-stage retrofit: (1) a script-prune crop that reduces 200,019 tokens to 131,072 by removing nine out-of-scope writing systems, and (2) a surgical retrofit of 2,372 corpus-dead vocabulary slots determined by linear-programming allocation across nine Brahmic Unicode blocks. The pre-tokenizer, decoder, and inherited merge rules are unchanged from o200k_base, making BrahmicTokenizer-131K a drop-in replacement at the tokenizer interface. On 27 million documents of public Indic pretraining text (2.84 billion words, 46.21 GB), BrahmicTokenizer-131K produces 26.7% fewer tokens than Mistral-Nemo Tekken / Sarvam-m at the same vocabulary budget, with per-language savings of 15.79% (Tamil) to 76.79% (Odia, a 4.31x compression ratio). The Odia advantage is mechanistically explained by Tekken/Sarvam-m containing zero Oriya-block tokens; our surgery added 725. On non-Indic content, BrahmicTokenizer-131K matches o200k_base's English fertility (1.235 vs 1.232 tokens/word) and beats Tekken/Sarvam-m by 4.0-14.2% on HumanEval, MBPP, and GSM8K. Across our 14-tokenizer benchmark, it is the only tokenizer simultaneously competitive on Brahmic, English, EU, code, and math at the 131K budget. Specialist tokenizers at other vocab classes (Sarvam-30B, Sarvam-1, MUTANT-Indic) achieve better Indic compression at the cost of non-Indic performance: Sarvam-1's English fertility is 15.9% worse and its code/math compression 26-33% worse than ours. We release the artifact under Apache 2.0 at https://huggingface.co/theschoolofai/BrahmicTokenizer-131K.

Related papers

• Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition [0.0]
  On identical FLORES-200 devtest content, Sanskrit costs 1.774-2.187 times the English tokens under deployed tokenizers with vocabularies of 200,019 ids or more, but only 1.325-1.353 times the Hindi tokens.<n>Against a deployed English tokenizer, Sanskrit-trained BPE arms then look cheaper per proposition than English on contemporary prose (0.887).<n>At 32,000 and 64,000 pieces all 8 matched pairs, each size-matched arm against both a pair-matched and a byte-matched control, sit above 1.0 on prose with 95% intervals excluding it.
  arXiv  Detail & Related papers  (2026-09-11T15:19:48Z)
• Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility [0.0]
  In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign.<n>Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition.<n>Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai)<n>For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali.
  arXiv  Detail & Related papers  (2026-08-26T22:56:24Z)
• Morphology Aware Reversible Semantic Tokenization and Hierarchical Word Composition for Tamil Language Models [0.0]
  We present a Tamil morphology system extending the open-source ThamizhiMorph analyzer and generator.<n>We compare a flat morphology tokenizer, a signal-preserving word composer, and tokenizers based on Sarvam-1, AI4Bharat IndicBERTv2, and BrahmicTokenizer-131K.<n> explicit Tamil morphology improves translation under a fixed small-model budget, while hierarchical composition substantially reduces sequence length and estimated inference cost.
  arXiv  Detail & Related papers  (2026-08-02T11:07:53Z)
• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages [0.0]
  Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.<n>We quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus.<n>We show that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents.
  arXiv  Detail & Related papers  (2026-07-27T11:20:29Z)
• Separate Before You Compress: The WWHO Tokenization Architecture [0.0]
  Current Large Language Models (LLMs) mostly use BPE (Byte Pair Linguist) based tokenizers.<n>We propose a new three-layer architecture, WWHO (Where-What-How), and an algorithm named SGPE (Syllable Grapheme Pair Linguist)<n>Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set.
  arXiv  Detail & Related papers  (2026-03-26T10:56:48Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece [0.0]
  Sanskrit is hypothesized to express more meaning per token due to its morphology and grammar rules.<n>We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi.<n>We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT.
  arXiv  Detail & Related papers  (2026-01-05T15:23:50Z)
• AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3 [4.284434049360481]
  We present AraToken, an Arabic-optimized tokenizer built on SentencePiece Unigram algorithm.<n>We show that SentencePiece with normalization achieves 18% lower fertility (1.199 vs 1.35 tokens/word) compared to unnormalized baselines.<n>Our experiments show that LEP reduces evaluation loss from 8.28 to 2.43 within 800 training steps on 100K Arabic samples.
  arXiv  Detail & Related papers  (2025-12-20T15:32:10Z)
• Explaining and Mitigating Crosslingual Tokenizer Inequities [18.823984182115712]
  Having high token premiums leads to less throughput during training and increases costs at inference.<n>We train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages.<n>We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do.
  arXiv  Detail & Related papers  (2025-10-24T17:36:03Z)
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [53.22544362024936]
  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.<n>Standard algorithms for learning tokenizers rely on frequency-based objectives.<n>We introduce Parity-aware Byte Pair.<n>We find empirically that Parity-aware BPE leads to more equitable token counts across languages.
  arXiv  Detail & Related papers  (2025-08-06T18:14:43Z)
• Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier [11.131061652781192]
  Pre-tokenization causes the distribution of tokens in a corpus to skew towards common, full-length words.<n>We propose BoundlessB, a modified BPE algorithm that relaxes the pretoken boundary constraint.<n>Our approach selectively merges two complete pretokens into a larger unit we term a superword.
  arXiv  Detail & Related papers  (2025-03-31T19:36:29Z)
• SuperBPE: Space Travel for Language Models [103.09169510391972]
  We introduce a "superword" tokenizer, SuperBPE, which incorporates a simple pretokenization curriculum into the byte-pair encoding (BPE) algorithm.<n>SuperBPE encodes a fixed piece of text with up to 33% fewer tokens than BPE on average.<n>Our model trained with SuperBPE achieves an average +4.0% absolute improvement over the BPE baseline across 30 downstream tasks.
  arXiv  Detail & Related papers  (2025-03-17T17:53:23Z)
• On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation [104.85258654917297]
  We find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance. We propose Language Aware Vocabulary Sharing (LAVS) to construct the multilingual vocabulary. We conduct experiments on a multilingual machine translation benchmark in 11 languages.
  arXiv  Detail & Related papers  (2023-05-18T12:43:31Z)
• Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers [48.036317742487796]
  We propose a new approach to tokenization for lexical matching retrieval algorithms. We use the WordPiece tokenizer, which can be built automatically from unsupervised data. Results show that the mBERT tokenizer provides strong relevance signals for retrieval "out of the box", outperforming whitespace tokenization on most languages.
  arXiv  Detail & Related papers  (2022-10-11T14:32:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.