Token-Native Storage: Read and Write in your Agent's Language

• URL: http://arxiv.org/abs/2608.02376v2
• Date: Thu, 06 Aug 2026 13:36:52 GMT
• Title: Token-Native Storage: Read and Write in your Agent's Language
• Abstract summary: We argue for token-native storage of text as the model's own byte-pair-encoding (BPE) token IDs.<n>BPE numbers tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer (streamvbyte) recover most of the entropy coder's ratio.<n>Because a model reads token IDs, not text, a token-native store hands over the IDs directly instead of retokenizing on every read.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Search and database engines still store text as UTF-8, a format built for humans. But the systems that increasingly read and write that text (embedders, rerankers, and language-model agents) work with token IDs, not characters, so every access pays to translate between the two. As agents become the primary readers and writers of stored text, we argue for token-native storage: keep the text as the model's own byte-pair-encoding (BPE) token IDs. Packing r50k IDs as uint16 already beats UTF-8 by 2.25x on English with no compression, and an entropy coder on top reaches 3.30x. Across six tokenizers and three corpora (English, code, Hindi), compressing token IDs matches or beats every byte codec, even a corpus-trained zstd dictionary. Two findings sharpen the case. BPE numbers tokens by merge order instead of frequency, and re-ranking by frequency lets a plain integer codec (streamvbyte) recover most of the entropy coder's ratio while decoding ~7x faster, a near-free change to how AI labs publish vocabularies. And because a model reads token IDs, not text, a token-native store hands over the IDs directly instead of re-tokenizing on every read. The only requirement is that reader and writer share a tokenizer, and different model families often use different ones today, so we argue for standardization: a published, shared vocabulary, the way ASCII and UTF-8 standardized text.

Related papers

• Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content [11.09559407806515]
  We ask what each encoding preserves when both the content and the downstream capacity are controlled.<n>Using verified parallel sentences across thirteen languages and five scripts, we compare tokens, bytes, and pixels through a shared bottleneck.<n>We evaluate three utilities: surface form preservation, cross-lingual sentence alignment, and topic classification.
  arXiv  Detail & Related papers  (2026-07-17T16:55:27Z)
• Frequency-Ordered Tokenization for Better Text Compression [0.0]
  Method tokenizes text with Byte Pair.<n> encodes the result with variable-length integers before passing it to any standard compressor.<n>On enwik8 (100 MB Wikipedia), this yields improvements of 7.08 percentage points (pp) for zlib, 1.69 pp for LZMA, and 0.76 pp for zstd.
  arXiv  Detail & Related papers  (2026-02-26T12:53:48Z)
• BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition [18.054653205555862]
  Byte-level BPE (BBPE) using Longer-8 is widely adopted for its language-agnostic design and full coverage.<n>We propose BBPE16, a Unicode-16-based BBPE tokenizer that represents most modern scripts with a uniform 2-byte code unit.
  arXiv  Detail & Related papers  (2026-02-02T06:56:27Z)
• UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8 [11.900027560734145]
  Subword tokenization segments input text according to a pre-defined vocabulary to feed it into a language model.<n>Using code points means that all members incrementally are valid-8 characters.<n>Beginning with bytes avoids out-of-vocabulary errors with only 256 initial members.
  arXiv  Detail & Related papers  (2025-11-05T00:08:01Z)
• Back to Bytes: Revisiting Tokenization Through UTF-8 [25.766926865651936]
  We present a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the underlying text's-8-8 encoding.<n>Our implementation never introduces out-of-range IDs.
  arXiv  Detail & Related papers  (2025-10-19T20:06:12Z)
• SuperBPE: Space Travel for Language Models [103.09169510391972]
  We introduce a "superword" tokenizer, SuperBPE, which incorporates a simple pretokenization curriculum into the byte-pair encoding (BPE) algorithm.<n>SuperBPE encodes a fixed piece of text with up to 33% fewer tokens than BPE on average.<n>Our model trained with SuperBPE achieves an average +4.0% absolute improvement over the BPE baseline across 30 downstream tasks.
  arXiv  Detail & Related papers  (2025-03-17T17:53:23Z)
• From Language Models over Tokens to Language Models over Characters [54.123846188068384]
  Modern language models are internally -- and mathematically -- distributions over $ittoken$ strings rather than $itcharacter$ strings.<n>This paper presents algorithms for converting token-level language models to character-level ones.
  arXiv  Detail & Related papers  (2024-12-04T21:19:20Z)
• Batching BPE Tokenization Merges [55.2480439325792]
  BatchBPE is an open-source pure Python implementation of the Byte Pair algorithm. It is used to train a high quality tokenizer on a basic laptop.
  arXiv  Detail & Related papers  (2024-08-05T09:37:21Z)
• T2S-GPT: Dynamic Vector Quantization for Autoregressive Sign Language Production from Text [59.57676466961787]
  We propose a novel dynamic vector quantization (DVA-VAE) model that can adjust the encoding length based on the information density in sign language. Experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. We propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts.
  arXiv  Detail & Related papers  (2024-06-11T10:06:53Z)
• Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal [58.29382184006158]
  We propose Scaffold-BPE, which incorporates a dynamic scaffold token removal mechanism by parameter-free, computation-light, and easy-to-implement modifications to the original BPE method. On extensive experiments across language modeling and even machine translation, Scaffold-BPE consistently outperforms the original BPE.
  arXiv  Detail & Related papers  (2024-04-27T07:12:07Z)
• Neural Machine Translation without Embeddings [44.129310924201604]
  Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via-8. Experiments on byteto-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models.
  arXiv  Detail & Related papers  (2020-08-21T09:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.