Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition

• URL: http://arxiv.org/abs/2609.12960v1
• Date: Fri, 11 Sep 2026 15:19:48 GMT
• Title: Fewer Words, Not Fewer Tokens: Measuring the Sanskrit Tokenization Penalty per Proposition
• Abstract summary: On identical FLORES-200 devtest content, Sanskrit costs 1.774-2.187 times the English tokens under deployed tokenizers with vocabularies of 200,019 ids or more, but only 1.325-1.353 times the Hindi tokens.<n>Against a deployed English tokenizer, Sanskrit-trained BPE arms then look cheaper per proposition than English on contemporary prose (0.887).<n>At 32,000 and 64,000 pieces all 8 matched pairs, each size-matched arm against both a pair-matched and a byte-matched control, sit above 1.0 on prose with 95% intervals excluding it.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sanskrit fuses case, number, person and tense into word endings and chains clauses into compounds, so it is information-dense per word. Whether that density survives subword tokenization is a separate question, to be asked per unit of meaning rather than per word. On identical FLORES-200 devtest content, Sanskrit costs 1.774-2.187 times the English tokens under deployed tokenizers with vocabularies of 200,019 ids or more, but only 1.325-1.353 times the Hindi tokens. Against a deployed English tokenizer, Sanskrit-trained BPE arms then look cheaper per proposition than English on contemporary prose (0.887). Against a matched English control, the same algorithm and vocabulary trained on the English side of the same corpus, that flip disappears: at 32,000 and 64,000 pieces all 8 matched pairs, each size-matched arm against both a pair-matched and a byte-matched control, sit above 1.0 on prose with 95% intervals excluding it. The gap closes as the vocabulary grows: at 128,000 pieces the BPE pair reads 0.983 in domain while staying above parity out of domain (1.025) and on FLORES (1.116). The ratio factorises into a character-length ratio and a tokens-per-character ratio, the second near 1 throughout: what survives matched tokenization is character-level length, which Sanskrit prose lacks over English in SLP1 (1.028) and Sanskrit verse has (0.596). The robust statement is about deployed practice: on contemporary prose and on FLORES, with the Sanskrit side in SLP1 against the deployed o200k English pivot, Sanskrit costs 1.831-2.899 English tokens per proposition under the tokenizers people actually ship. Code, the results snapshot and every table here are public.

Related papers

• Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility [0.0]
  In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign.<n>Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition.<n>Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai)<n>For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali.
  arXiv  Detail & Related papers  (2026-08-26T22:56:24Z)
• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages [0.0]
  Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.<n>We quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus.<n>We show that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents.
  arXiv  Detail & Related papers  (2026-07-27T11:20:29Z)
• BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base [0.0]
  BrahmicTokenizer-131K is a 131,072-vocabulary byte-level BPE tokenizer.<n>It closes the Brahmic compression gap at the 131K-vocabulary class.<n>It preserves the English, EU-language, and code compression of OpenAI's o200k_base.
  arXiv  Detail & Related papers  (2026-05-28T05:29:12Z)
• Separate Before You Compress: The WWHO Tokenization Architecture [0.0]
  Current Large Language Models (LLMs) mostly use BPE (Byte Pair Linguist) based tokenizers.<n>We propose a new three-layer architecture, WWHO (Where-What-How), and an algorithm named SGPE (Syllable Grapheme Pair Linguist)<n>Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set.
  arXiv  Detail & Related papers  (2026-03-26T10:56:48Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece [0.0]
  Sanskrit is hypothesized to express more meaning per token due to its morphology and grammar rules.<n>We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi.<n>We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT.
  arXiv  Detail & Related papers  (2026-01-05T15:23:50Z)
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• Beyond Literal Token Overlap: Token Alignability for Multilinguality [53.680462160878925]
  We propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation.<n>In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low.<n>We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2025-02-10T13:50:12Z)
• Translate to Disambiguate: Zero-shot Multilingual Word Sense Disambiguation with Pretrained Language Models [67.19567060894563]
  Pretrained Language Models (PLMs) learn rich cross-lingual knowledge and can be finetuned to perform well on diverse tasks. We present a new study investigating how well PLMs capture cross-lingual word sense with Contextual Word-Level Translation (C-WLT) We find that as the model size increases, PLMs encode more cross-lingual word sense knowledge and better use context to improve WLT performance.
  arXiv  Detail & Related papers  (2023-04-26T19:55:52Z)
• Phonotactic Complexity and its Trade-offs [73.10961848460613]
  This simple measure allows us to compare the entropy across languages. We demonstrate a very strong negative correlation of -0.74 between bits per phoneme and the average length of words.
  arXiv  Detail & Related papers  (2020-05-07T21:36:59Z)
• Robust Cross-lingual Embeddings from Parallel Sentences [65.85468628136927]
  We propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word representations. Our approach significantly improves crosslingual sentence retrieval performance over all other approaches. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task.
  arXiv  Detail & Related papers  (2019-12-28T16:18:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.