Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

• URL: http://arxiv.org/abs/2606.14122v1
• Date: Fri, 12 Jun 2026 05:03:55 GMT
• Title: Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
• Abstract summary: Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid-8 sequences when encountering rare or unseen characters.<n>We investigate the relationship between training scale and-8 generation reliability with a 355 parameter model trained on 80B tokens from a balanced multilingual corpus of English, Japanese, and Chinese.<n>In context-free generation, rare characters achieve higher structural validity than common characters, suggesting over-specialization of frequent character representations.
• Score: 20.9586694123981
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investigate the relationship between training scale and UTF-8 generation reliability with a 355M parameter model trained on 80B tokens from a balanced multilingual corpus of English, Japanese, Korean, and Chinese. We introduce multiple evaluation protocols that isolate UTF-8 structural validity from language modeling. UTF-8 validity convergence lags perplexity by a roughly a factor of two: perplexity stabilizes after 2.1B tokens, but UTF-8 validity requires 4.2B tokens. In context-free generation, rare characters achieve higher structural validity than common characters, suggesting over-specialization of frequent character representations. Through experiments, we observed that reliable UTF-8 generation is a distinct capability requiring evaluation beyond perplexity.

Related papers

• BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition [18.054653205555862]
  Byte-level BPE (BBPE) using Longer-8 is widely adopted for its language-agnostic design and full coverage.<n>We propose BBPE16, a Unicode-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit.
  arXiv  Detail & Related papers  (2026-02-02T06:56:27Z)
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8 [11.900027560734145]
  Subword tokenization segments input text according to a pre-defined vocabulary to feed it into a language model.<n>Using code points means that all members incrementally are valid-8 characters.<n>Beginning with bytes avoids out-of-vocabulary errors with only 256 initial members.
  arXiv  Detail & Related papers  (2025-11-05T00:08:01Z)
• Back to Bytes: Revisiting Tokenization Through UTF-8 [25.766926865651936]
  We present a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the underlying text's-8-8 encoding.<n>Our implementation never introduces out-of-range IDs.
  arXiv  Detail & Related papers  (2025-10-19T20:06:12Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Sampling from Your Language Model One Byte at a Time [82.71473348639489]
  Tokenization can introduce distortion into the model's generations, known as the Prompt Boundary Problem (PBP)<n>We present an inference-time method to convert any autore LM with a BPE tokenizer into a character-level or byte-level LM.<n>Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers.
  arXiv  Detail & Related papers  (2025-06-17T02:37:04Z)
• Language Models over Canonical Byte-Pair Encodings [56.09166157337198]
  We propose methods to enforce canonicality in token-level language models.<n>We show that fixing canonicality mistakes improves the likelihood of held-out data for several models and corpora.
  arXiv  Detail & Related papers  (2025-06-09T17:26:14Z)
• From Language Models over Tokens to Language Models over Characters [54.123846188068384]
  Modern language models are internally -- and mathematically -- distributions over $ittoken$ strings rather than $itcharacter$ strings.<n>This paper presents algorithms for converting token-level language models to character-level ones.
  arXiv  Detail & Related papers  (2024-12-04T21:19:20Z)
• Signs as Tokens: A Retrieval-Enhanced Multilingual Sign Language Generator [55.94334001112357]
  We introduce a multilingual sign language model, Signs as Tokens (SOKE), which can generate 3D sign avatars autoregressively from text inputs.<n>We propose a retrieval-enhanced SLG approach, which incorporates external sign dictionaries to provide accurate word-level signs.
  arXiv  Detail & Related papers  (2024-11-26T18:28:09Z)
• Local Byte Fusion for Neural Machine Translation [19.16966721276286]
  Subword tokenization schemes are the dominant technique used in current NLP models. Byte-based methods i.e. tokenization into byte sequences are an alternative. Experiments on multilingual translation, zero-shot cross-lingual transfer, and domain adaptation reveal a consistent improvement over traditional models.
  arXiv  Detail & Related papers  (2022-05-23T17:49:02Z)
• Zero-Shot Cross-lingual Semantic Parsing [56.95036511882921]
  We study cross-lingual semantic parsing as a zero-shot problem without parallel data for 7 test languages. We propose a multi-task encoder-decoder model to transfer parsing knowledge to additional languages using only English-Logical form paired data. Our system frames zero-shot parsing as a latent-space alignment problem and finds that pre-trained models can be improved to generate logical forms with minimal cross-lingual transfer penalty.
  arXiv  Detail & Related papers  (2021-04-15T16:08:43Z)
• Neural Machine Translation without Embeddings [44.129310924201604]
  Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via-8. Experiments on byteto-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models.
  arXiv  Detail & Related papers  (2020-08-21T09:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.