SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

• URL: http://arxiv.org/abs/2508.11857v2
• Date: Mon, 25 Aug 2025 13:30:15 GMT
• Title: SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
• Abstract summary: Tokenization remains a fundamental yet underexplored bottleneck in natural language processing.<n>We present SupraTok, a novel tokenization architecture that reimagines subword segmentation.<n>Our approach achieves 31% improvement in English tokenization efficiency.
• Score: 1.9336815376402718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tokenization remains a fundamental yet underexplored bottleneck in natural language processing, with strategies largely static despite remarkable progress in model architectures. We present SupraTok, a novel tokenization architecture that reimagines subword segmentation through three innovations: cross-boundary pattern learning that discovers multi-word semantic units, entropy-driven data curation that optimizes training corpus quality, and multi-phase curriculum learning for stable convergence. Our approach extends Byte-Pair Encoding by learning "superword" tokens, coherent multi-word expressions that preserve semantic unity while maximizing compression efficiency. SupraTok achieves 31% improvement in English tokenization efficiency (5.91 versus 4.51 characters per token) compared to OpenAI's o200k tokenizer and 30% improvement over Google's Gemma 3 tokenizer (256k vocabulary), while maintaining competitive performance across 38 languages. When integrated with a GPT-2 scale model (124M parameters) trained on 10 billion tokens from the FineWeb-Edu dataset, SupraTok yields 8.4% improvement on HellaSWAG and 9.5% on MMLU benchmarks without architectural modifications. While these results are promising at this scale, further validation at larger model scales is needed. These findings suggest that efficient tokenization can complement architectural innovations as a path to improved language model performance.

Related papers

• What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling? [1.3177681589844814]
  Tokenization is a fundamental component of language modeling pipelines.<n>In this work, we analyze what tokens are learned when tokenization is jointly optimized with language modeling.<n>We compare tokenizer-free approaches such as SSLMs and H-Nets with fixed tokenizers across 18 typologically and script-diverse languages.
  arXiv  Detail & Related papers  (2026-08-18T03:34:56Z)
• Adding Robust Code-Switching Capabilities to High Performance Multilingual ASR [61.09730719510015]
  We extend models to handle complex code-switching, including morphological variations across languages.<n>Our approach reduces transcription errors by 32.87% on code-switched words while improving overall WER by 5.31%, all while maintaining mono-lingual performance.
  arXiv  Detail & Related papers  (2026-06-20T11:02:29Z)
• $\mathcal{S}^2$IT: Stepwise Syntax Integration Tuning for Large Language Models in Aspect Sentiment Quad Prediction [51.165432266846096]
  syntactic structure information is underutilized in the generative paradigm of large language models (LLMs)<n>S2IT progressively integrates syntactic structure knowledge into LLMs through a multi-step tuning process.<n> Experiments demonstrate that S2IT significantly improves state-of-the-art performance across multiple datasets.
  arXiv  Detail & Related papers  (2026-04-25T13:24:29Z)
• Incentivizing Parametric Knowledge via Reinforcement Learning with Verifiable Rewards for Cross-Cultural Entity Translation [68.85147984815778]
  We propose EA-RLVR (Entity-Anchored Reinforcement Learning with Verifiable Rewards) to incentivize the effective use of parametric knowledge.<n>EA-RLVR anchors supervision on a verifiable, entity-level reward signal and incorporates lightweight structural gates to stabilize optimization.<n>We evaluate EA-RLVR on XC-Translate and observe consistent improvements in both entity translation accuracy and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-04-18T07:15:43Z)
• Optimizing Korean-Centric LLMs via Token Pruning [6.029880646740327]
  token pruning is a compression technique that eliminates tokens and embedding parameters corresponding to languages irrelevant to the target application.<n>We evaluate architectures including Qwen3, Gemma-3, Llama-3, and Aya across three vocabulary configurations.
  arXiv  Detail & Related papers  (2026-04-17T16:53:14Z)
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging [65.07273789940116]
  This paper introduces a hierarchical architecture that jointly optimize a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks.<n> MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation.
  arXiv  Detail & Related papers  (2025-11-17T19:27:41Z)
• IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs [5.068673710249497]
  IndicSuperTokenizer is a tokenizer for Indic multilingual LLMs.<n>It combines subword and multi-word tokenization, along with language-specific tokens pre-tokenization.<n>It improves the average fertility score by 39.5% over LLaMA4 and by 18% over Sutra.
  arXiv  Detail & Related papers  (2025-11-05T06:57:42Z)
• CS3-Bench: Evaluating and Enhancing Speech-to-Speech LLMs for Mandarin-English Code-Switching [31.584937435966253]
  We propose Code-Switching Speech-to-Speech Benchmark (CS3-Bench)<n>Experiments on 7 mainstream models demonstrate a relative performance drop of up to 66% in knowledge-intensive question answering.<n>Our approach improves the knowledge accuracy from 25.14% to 46.13%, with open-ended understanding rate from 64.5% to 86.5%, and significantly reduces pronunciation errors in the secondary language.
  arXiv  Detail & Related papers  (2025-10-09T07:34:23Z)
• The Art of Breaking Words: Rethinking Multilingual Tokenizer Design [21.9940001977516]
  Existing tokenizers exhibit high token-to-word ratios, inefficient use of context length, and slower inference.<n>We present a systematic study that links vocabulary size, pre-tokenization rules, and training-corpus composition to both token-to-word efficiency and model quality.<n>Our tokenizer achieves more than 40% improvement on average token-to-word ratio against stateof-the-art multilingual Indic models.
  arXiv  Detail & Related papers  (2025-08-03T15:31:10Z)
• Improving Contextual ASR via Multi-grained Fusion with Large Language Models [12.755830619473368]
  We propose a novel multi-grained fusion approach that jointly leverages the strengths of both token-level and phrase-level fusion with Large Language Models (LLMs)<n>Our approach incorporates a late-fusion strategy that combines ASR's acoustic information with LLM's rich contextual knowledge, balancing fine-grained token precision with holistic phrase-level understanding.<n> Experiments on Chinese and English datasets demonstrate that our approach achieves state-of-the-art performance on keyword-related metrics.
  arXiv  Detail & Related papers  (2025-07-16T13:59:32Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training [57.62126373849383]
  Cross-lingual In-context Pre-training (CrossIC-PT) is a simple and scalable approach that enhances cross-lingual transfer.<n>We construct CrossIC-PT samples by interleaving semantic-related bilingual Wikipedia documents into a single context window.<n> Experimental results demonstrate that CrossIC-PT improves multilingual performance on three models across six target languages.
  arXiv  Detail & Related papers  (2025-04-29T07:24:25Z)
• Learning Adaptive Parallel Reasoning with Language Models [70.1745752819628]
  We propose Adaptive Parallel Reasoning (APR), a novel reasoning framework that enables language models to orchestrate both serialized and parallel computations end-to-end.<n> APR generalizes existing reasoning methods by enabling adaptive multi-threaded inference using spawn() and join() operations.<n>A key innovation is our end-to-end reinforcement learning strategy, optimizing both parent and child inference threads to enhance task success rate without requiring predefined reasoning structures.
  arXiv  Detail & Related papers  (2025-04-21T22:29:02Z)
• Towards Typologically Aware Rescoring to Mitigate Unfaithfulness in Lower-Resource Languages [9.426642998924724]
  Multilingual large language models generate non-faithful output in resource-constrained languages.<n>To mitigate unfaithfulness in such settings, we propose using computationally light auxiliary models to rescore the outputs of larger architectures.<n>We show that monolingual 4-layer BERT models pretrained from scratch on less than 700 MB of data without fine-tuning are able to identify faithful summaries with a mean accuracy of 88.33%.
  arXiv  Detail & Related papers  (2025-02-24T21:22:19Z)
• Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark [0.29687381456163997]
  Tokenization is a fundamental preprocessing step in NLP, directly impacting large language models' ability to capture syntactic, morphosyntactic, and semantic structures.<n>This paper introduces a novel framework for evaluating tokenization strategies, addressing challenges in morphologically rich and low-resource languages.
  arXiv  Detail & Related papers  (2025-02-10T21:47:49Z)
• SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator [65.62084602011596]
  Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks.<n>We have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens.<n>We introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens.
  arXiv  Detail & Related papers  (2024-12-16T18:58:57Z)
• PanGu-{\Sigma}: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing [64.53242758625922]
  PanGu-Sigma is trained on a cluster of Ascend 910 AI processors and MindSpore framework. It provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks.
  arXiv  Detail & Related papers  (2023-03-20T03:39:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.