Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece

• URL: http://arxiv.org/abs/2601.06142v1
• Date: Mon, 05 Jan 2026 15:23:50 GMT
• Title: Is Sanskrit the most token-efficient language? A quantitative study using GPT, Gemini, and SentencePiece
• Abstract summary: Sanskrit is hypothesized to express more meaning per token due to its morphology and grammar rules.<n>We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi.<n>We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tokens are the basic units of Large Language Models (LLMs). LLMs rely on tokenizers to segment text into these tokens, and tokenization is the primary determinant of computational and inference cost. Sanskrit, one of the oldest languages, is hypothesized to express more meaning per token due to its morphology and grammar rules; however, no prior work has quantified this. We use a dataset of 701 parallel verses of the Bhagavad Gita, which comprises three languages-Sanskrit, English, and Hindi along with transliteration of Sanskrit into English. We test tokenizers including SentencePiece (SPM), older GPT models, and the latest generation tokenizers from Gemini and GPT. We use metrics of token count, characters per token (token efficiency), and tokens per character (token cost). Results show a ~2x difference in token counts between Sanskrit and English/Hindi under the unbiased SPM baseline. English/Hindi translations of Sanskrit commentary resulted in an approximately 20x increase in token count. GPT o200k base (latest, used by GPT-4o) and Gemini (latest) reduce bias by a significant degree compared to GPT cl100k base (used until GPT-4), but still fail to fully capture Sanskrit's compactness. This matters because there might be a penalty bias for non-English users, which inflates the token count. This research provides a foundation for improving future tokenizer design and shows the potential of Sanskrit for highly compact encoding, saving on cost while speeding up training and inference. The code and dataset are available at https://github.com/anshulkr713/sanskrit-token-efficiency

Related papers

• Measuring the Tokenization Premium: A Cost Audit for Underserved Language Communities [0.0]
  We introduce the Tokenization Equity Audit (TEA), a benchmark for measuring tokenization premiums in technical tutoring content.<n>Tea evaluates three widely used tokenizers, GPT-4o's o200k base, Qwen2.5-7B, and Mistral-7B, on a 120-item Python corpus translated from English into Bengali, Hindi, Arabic, Tamil, and Yoruba.<n> Bengali requires (1.56times) as many GPT-4o tokens as English, reducing a nominal 128k-token context window to an effective 82k-equivalent English-token capacity for the same semantic content.
  arXiv  Detail & Related papers  (2026-08-10T02:51:04Z)
• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages [0.0]
  Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.<n>We quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus.<n>We show that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents.
  arXiv  Detail & Related papers  (2026-07-27T11:20:29Z)
• Separate Before You Compress: The WWHO Tokenization Architecture [0.0]
  Current Large Language Models (LLMs) mostly use BPE (Byte Pair Linguist) based tokenizers.<n>We propose a new three-layer architecture, WWHO (Where-What-How), and an algorithm named SGPE (Syllable Grapheme Pair Linguist)<n>Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set.
  arXiv  Detail & Related papers  (2026-03-26T10:56:48Z)
• Reducing Tokenization Premiums for Low-Resource Languages [5.02470728447561]
  Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs.<n>This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages.<n>We propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens.
  arXiv  Detail & Related papers  (2026-01-19T19:08:58Z)
• Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE [0.0]
  BengaliBPE is a language-aware subword tokenizer for the Bengali script.<n>It applies Unicode normalization and morphology-aware merge rules to maintain linguistic consistency and preserve subword integrity.<n>It provides the most detailed segmentation and the best morphological interpretability, albeit with slightly higher computational cost.
  arXiv  Detail & Related papers  (2025-11-07T15:23:32Z)
• Explaining and Mitigating Crosslingual Tokenizer Inequities [18.823984182115712]
  Having high token premiums leads to less throughput during training and increases costs at inference.<n>We train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages.<n>We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do.
  arXiv  Detail & Related papers  (2025-10-24T17:36:03Z)
• Speculating LLMs' Chinese Training Data Pollution from Their Tokens [59.325920108736454]
  Polluted Chinese (PoC) tokens are indicated contents like pornography or online gambling.<n>We give a formal definition and taxonomy of PoC tokens based on the GPT's vocabulary.<n>We build a PoC token detector via fine-tuning an LLM to label PoC tokens in vocabularies.
  arXiv  Detail & Related papers  (2025-08-25T08:08:51Z)
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP [0.2826977330147589]
  Tokenization plays a pivotal role in natural language processing (NLP)<n>We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation.<n>The method uses phono normalization, root-affix, and a novel algorithm that balances morpheme preservation with vocabulary efficiency.
  arXiv  Detail & Related papers  (2025-08-19T22:17:42Z)
• Causal Estimation of Tokenisation Bias [58.20086589761273]
  We quantify the effect of including or not a subword in a tokeniser's vocabulary on the probability a trained model assigns to the corresponding characters.<n>We find that tokenisation consistently affects models' outputs across scales, vocabularies, and tokenisers.<n> Notably, a subword's presence in a small model's vocabulary may increase its characters' probability by up to 17 times.
  arXiv  Detail & Related papers  (2025-06-03T17:59:47Z)
• Beyond Literal Token Overlap: Token Alignability for Multilinguality [53.680462160878925]
  We propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation.<n>In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low.<n>We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2025-02-10T13:50:12Z)
• From Language Models over Tokens to Language Models over Characters [54.123846188068384]
  Modern language models are internally -- and mathematically -- distributions over $ittoken$ strings rather than $itcharacter$ strings.<n>This paper presents algorithms for converting token-level language models to character-level ones.
  arXiv  Detail & Related papers  (2024-12-04T21:19:20Z)
• Egalitarian Language Representation in Language Models: It All Begins with Tokenizers [0.0]
  We show that not all tokenizers offer fair representation for complex script languages such as Tamil, Sinhala, and Hindi. We introduce an improvement to the Byte Pair algorithm by incorporating graphemes, which we term Grapheme Pair. Our experiments show that grapheme-based character extraction outperforms byte-level tokenizers for complex scripts.
  arXiv  Detail & Related papers  (2024-09-17T19:05:37Z)
• CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM [7.6361655389908965]
  CharPoet is a Chinese classical poetry generation system based on token-free LLM. Our token-free architecture generates in a character-by-character manner, enabling precise control over the number of characters. CharPoet achieves format accuracy above 0.96, outperforming Jiuge-GPT-2 (0.91) and GPT-4 (0.38)
  arXiv  Detail & Related papers  (2024-01-07T15:00:36Z)
• Better Than Whitespace: Information Retrieval for Languages without Custom Tokenizers [48.036317742487796]
  We propose a new approach to tokenization for lexical matching retrieval algorithms. We use the WordPiece tokenizer, which can be built automatically from unsupervised data. Results show that the mBERT tokenizer provides strong relevance signals for retrieval "out of the box", outperforming whitespace tokenization on most languages.
  arXiv  Detail & Related papers  (2022-10-11T14:32:46Z)
• SHUOWEN-JIEZI: Linguistically Informed Tokenizers For Chinese Language Model Pretraining [48.880840711568425]
  We study the influences of three main factors on the Chinese tokenization for pretrained language models. We propose three kinds of tokenizers: SHUOWEN (meaning Talk Word), the pronunciation-based tokenizers; 2) JIEZI (meaning Solve Character), the glyph-based tokenizers. We find that SHUOWEN and JIEZI tokenizers can generally outperform conventional single-character tokenizers.
  arXiv  Detail & Related papers  (2021-06-01T11:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.