Back to Bytes: Revisiting Tokenization Through UTF-8

• URL: http://arxiv.org/abs/2510.16987v1
• Date: Sun, 19 Oct 2025 20:06:12 GMT
• Title: Back to Bytes: Revisiting Tokenization Through UTF-8
• Abstract summary: We present a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the underlying text's-8-8 encoding.<n>Our implementation never introduces out-of-range IDs.
• Score: 25.766926865651936
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present UTF8Tokenizer, a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the bytes underlying the text's UTF-8 encoding (e.g., byte x09 is token ID 9). Unlike prior byte-level approaches (Xue et al., 2021; Pagnoni et al., 2025), our implementation never introduces out-of-range IDs (i.e. there is no token ID 256) or auxiliary tokens: all special behavior (e.g., padding, boundaries, conversation structure, attention segments, tool calling, "thinking" spans, etc.) is encoded using C0 control bytes - just as ASCII was originally designed to embed control information alongside printable text. These design principles yield practical benefits: (1) faster tokenization (14x) and significantly lower host-device transfer (8x less than int64); (2) simple, shareable 256*d embedding tables that can be aligned across models; and (3) a training-time enhancement via bit-biased embeddings, which exposes per-byte bit structure and can be added to the embedding table post-training, removing inference costs. Our HuggingFace-compatible implementation improves language modeling convergence.

Related papers

• Token-Native Storage: Read and Write in your Agent's Language [0.0]
  We argue for token-native storage of text as the model's own byte-pair-encoding (BPE) token IDs.<n>BPE numbers tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer (streamvbyte) recover most of the entropy coder's ratio.<n>Because a model reads token IDs, not text, a token-native store hands over the IDs directly instead of retokenizing on every read.
  arXiv  Detail & Related papers  (2026-08-03T15:20:36Z)
• BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition [18.054653205555862]
  Byte-level BPE (BBPE) using Longer-8 is widely adopted for its language-agnostic design and full coverage.<n>We propose BBPE16, a Unicode-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit.
  arXiv  Detail & Related papers  (2026-02-02T06:56:27Z)
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8 [11.900027560734145]
  Subword tokenization segments input text according to a pre-defined vocabulary to feed it into a language model.<n>Using code points means that all members incrementally are valid-8 characters.<n>Beginning with bytes avoids out-of-vocabulary errors with only 256 initial members.
  arXiv  Detail & Related papers  (2025-11-05T00:08:01Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization [0.0]
  SCRIPT (Script Category Representation in PreTokenization) is a novel encoding scheme that bypasses byte-8 conversion by using initial tokens based on Unicode script and category properties.<n>Our experiments demonstrate that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties for non-Latin-script languages.
  arXiv  Detail & Related papers  (2025-05-30T15:12:41Z)
• Language-Guided Image Tokenization for Generation [63.0859685332583]
  TexTok is a simple yet effective tokenization framework that leverages language to provide a compact, high-level semantic representation.<n>Compared to the conventional tokenizer without text conditioning, TexTok achieves average reconstruction FID improvements of 29.2% and 48.1% on ImageNet-256 and -512 benchmarks respectively.<n>TexTok with a vanilla DiT generator achieves state-of-the-art FID scores of 1.46 and 1.62 on ImageNet-256 and -512 respectively.
  arXiv  Detail & Related papers  (2024-12-08T03:18:17Z)
• Tokenization as Finite-State Transduction [24.19959327497118]
  We introduce a finite-state framework which can efficiently encode all possible tokenizations of a regular language. We show that Byte-Pair. Match (BPE) and MaxPiece (WordPiece) fit within this framework. An application of this is to guided generation, where the outputs of a language model are constrained to match some pattern.
  arXiv  Detail & Related papers  (2024-10-21T07:10:07Z)
• TULIP: Token-length Upgraded CLIP [57.818513403100326]
  We address the challenge of representing long captions in vision-language models, such as CLIP.<n>By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens.<n>We propose a generalizable method, named T, able to upgrade the token length to any length for CLIP-like models.
  arXiv  Detail & Related papers  (2024-10-13T22:34:15Z)
• Batching BPE Tokenization Merges [55.2480439325792]
  BatchBPE is an open-source pure Python implementation of the Byte Pair algorithm. It is used to train a high quality tokenizer on a basic laptop.
  arXiv  Detail & Related papers  (2024-08-05T09:37:21Z)
• SEP: Self-Enhanced Prompt Tuning for Visual-Language Model [93.94454894142413]
  We introduce a novel approach named Self-Enhanced Prompt Tuning (SEP) SEP explicitly incorporates discriminative prior knowledge to enhance both textual-level and visual-level embeddings. Comprehensive evaluations across various benchmarks and tasks confirm SEP's efficacy in prompt tuning.
  arXiv  Detail & Related papers  (2024-05-24T13:35:56Z)
• Local Byte Fusion for Neural Machine Translation [19.16966721276286]
  Subword tokenization schemes are the dominant technique used in current NLP models. Byte-based methods i.e. tokenization into byte sequences are an alternative. Experiments on multilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a consistent improvement over traditional models.
  arXiv  Detail & Related papers  (2022-05-23T17:49:02Z)
• Neural Machine Translation without Embeddings [44.129310924201604]
  Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via-8. Experiments on byteto-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models.
  arXiv  Detail & Related papers  (2020-08-21T09:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.