Separate Before You Compress: The WWHO Tokenization Architecture

• URL: http://arxiv.org/abs/2603.25309v1
• Date: Thu, 26 Mar 2026 10:56:48 GMT
• Title: Separate Before You Compress: The WWHO Tokenization Architecture
• Abstract summary: Current Large Language Models (LLMs) mostly use BPE (Byte Pair Linguist) based tokenizers.<n>We propose a new three-layer architecture, WWHO (Where-What-How), and an algorithm named SGPE (Syllable Grapheme Pair Linguist)<n>Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current Large Language Models (LLMs) mostly use BPE (Byte Pair Encoding) based tokenizers, which are very effective for simple structured Latin scripts such as English. However, standard BPE tokenizers struggle to process complex Abugida scripts due to their structural complexity. The problem is that these tokenizers break complex conjuncts, which are multi-codepoint grapheme clusters, into meaningless sub-character units. This degrades the LLM's reasoning efficiency by forcing it to learn basic orthographic structures at inference time and raises inference costs, resulting in a significant "Token Tax" for the Global South. We propose a new three-layer architecture, the WWHO (Where-What-How Often), and an algorithm named SGPE (Syllable-aware Grapheme Pair Encoding) that separates the linguistic rules of the script from the statistical compression process while enabling seamless multilingual tokenization. Using Sinhala and Devanagari (Hindi/Sanskrit) as highly complex Abugida scripts, we trained WWHO on a cleaned 30-million-sentence dataset and evaluated on a 1,499,950-sentence test set. For Sinhala, SGPE achieves a Token to Word Ratio (TWR) of 1.274 with 4.83 characters per token, representing a 61.7 percent reduction in tokens compared to OpenAI's o200k base. For Hindi, it achieves a TWR of 1.181 (27.0 percent reduction vs o200k). On the mixed-script (Sinhala, Devanagari, and English) dataset, SGPE achieves an overall TWR of 1.240, representing token reductions of 36.7 percent, 39.6 percent, and 60.2 percent relative to o200k base, Llama 4 Scout, and DeepSeek V3, respectively. This effectively extends the usable context window by up to 4.38 times for these Abugida languages while ensuring a Linguistic Zero-Breakage Guarantee, which ensures that no valid syllable is ever split across multiple tokens.

Related papers

• Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility [0.0]
  In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign.<n>Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition.<n>Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai)<n>For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali.
  arXiv  Detail & Related papers  (2026-08-26T22:56:24Z)
• The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages [0.0]
  Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words.<n>We quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus.<n>We show that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents.
  arXiv  Detail & Related papers  (2026-07-27T11:20:29Z)
• BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis [0.0]
  This paper presents BHARATI, a set of SentencePiece BPE token trained on a balanced 781 MB corpus spanning seven languages.<n>We describe three successive tokenizer versions: v1 (English and Sanskrit only), v2 (four-language support with byte-fallback for Tamil), and v3 (full seven-language native subword coverage)<n>Subword fertility analysis demonstrates that v3 averages 2.6 tokens per Indian Knowledge System (IKS) technical term, compared to 5.25 tokens per term with GPT-2's tokenizer and 3.75 tokens with the multilingual SentencePiece baseline.
  arXiv  Detail & Related papers  (2026-07-25T18:23:06Z)
• Compute Optimal Tokenization [79.3815358070537]
  We investigate how the information granularity of tokens, controlled by the compression rate, affects scaling trends.<n>We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate.<n>Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived.
  arXiv  Detail & Related papers  (2026-05-02T01:53:22Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers [76.59130257385826]
  Intermediate merge residues in BPE vocabularies are frequent during merge learning so that retained in the final vocabulary, but are mostly further merged and rarely emitted when tokenizing the corpus during tokenizer usage.<n>We present a systematic empirical characterization of this phenomenon across commonly used tokenizers and introduce LiteToken, a simple method for removing residue tokens.<n>Experiments show that LiteToken reduces token fragmentation, reduces parameters, and improves robustness to noisy or misspelled inputs, while preserving overall performance.
  arXiv  Detail & Related papers  (2026-02-04T16:19:05Z)
• The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models [0.0]
  We quantify a script tax by comparing two orthographic variants with identical linguistic content.<n>Across mBERT and XLM-R, the higher-fragmentation orthography shows a 3.4x increase in fertility.<n>Using bits per character (BPC) to avoid the "NLL paradox" from subword fragmentation, we find a substantial increase in information cost.
  arXiv  Detail & Related papers  (2026-01-19T14:45:40Z)
• AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3 [4.284434049360481]
  We present AraToken, an Arabic-optimized tokenizer built on SentencePiece Unigram algorithm.<n>We show that SentencePiece with normalization achieves 18% lower fertility (1.199 vs 1.35 tokens/word) compared to unnormalized baselines.<n>Our experiments show that LEP reduces evaluation loss from 8.28 to 2.43 within 800 training steps on 100K Arabic samples.
  arXiv  Detail & Related papers  (2025-12-20T15:32:10Z)
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP [0.2826977330147589]
  Tokenization plays a pivotal role in natural language processing (NLP)<n>We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation.<n>The method uses phono normalization, root-affix, and a novel algorithm that balances morpheme preservation with vocabulary efficiency.
  arXiv  Detail & Related papers  (2025-08-19T22:17:42Z)
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [53.22544362024936]
  Tokenization is the first -- and often least scrutinized -- step of most NLP pipelines.<n>Standard algorithms for learning tokenizers rely on frequency-based objectives.<n>We introduce Parity-aware Byte Pair.<n>We find empirically that Parity-aware BPE leads to more equitable token counts across languages.
  arXiv  Detail & Related papers  (2025-08-06T18:14:43Z)
• MorphTok: Morphologically Grounded Tokenization for Indian Languages [18.594241501479747]
  Tokenization is a crucial step in NLP, especially with the rise of large language models (LLMs)<n>We propose morphology-aware segmentation as a pre-tokenization step before applying the classical Byte-pair.<n>To handle the dependent vowels common in syllable-based writing systems, we propose Constrained BPE (CBPE)<n>CBPE handles dependent vowels to form a cohesive unit with other characters instead of occurring as a single unit.
  arXiv  Detail & Related papers  (2025-04-14T15:44:45Z)
• SuperBPE: Space Travel for Language Models [103.09169510391972]
  We introduce a "superword" tokenizer, SuperBPE, which incorporates a simple pretokenization curriculum into the byte-pair encoding (BPE) algorithm.<n>SuperBPE encodes a fixed piece of text with up to 33% fewer tokens than BPE on average.<n>Our model trained with SuperBPE achieves an average +4.0% absolute improvement over the BPE baseline across 30 downstream tasks.
  arXiv  Detail & Related papers  (2025-03-17T17:53:23Z)
• Vision-centric Token Compression in Large Language Model [51.92055188780033]
  Vision Centric Token Compression (Vist) is a slow-fast compression framework that mirrors human reading.<n>On eleven in-context learning benchmarks, Vist achieves the same accuracy with 2.3 times fewer tokens, cutting FLOPs by 16% and memory by 50%.
  arXiv  Detail & Related papers  (2025-02-02T13:10:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.