Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result

• URL: http://arxiv.org/abs/2608.28151v1
• Date: Fri, 28 Aug 2026 10:13:32 GMT
• Title: Nested Byte-Level Vocabularies Are Cheap to Deploy and Expensive to Share: A Pre-Registered Negative Result
• Abstract summary: We show how one language model can operate at several vocabulary sizes.<n>A control token indicates the active size, and be deployed at any trained size by slicing its embedding and output head.<n>A control with neither cap token nor output restriction is equally robust, attributing this benefit to multi-granularity training rather than conditioning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A byte-level BPE tokenizer is an ordered list of merge rules, so applying only a prefix yields a vocabulary whose token identifiers are the first rows of the full vocabulary. This prefix nesting allows one language model to operate at several vocabulary sizes, use a control token to indicate the active size, and be deployed at any trained size by slicing its embedding and output head. We pre-registered five claims, including margins, seeds, contrasts, and a stop rule, and trained 30 models with 3.1M- and 10.6M-parameter bodies on 200M tokens each. Slicing is numerically exact: across 76 checks, a sliced model reproduces the restricted full model's logits bit for bit and removes 66% of deployed weights without changing latency. However, the shared model trails a fixed-cap specialist by 3.64% bits per byte at 32k against a 1% margin, and by 2.96% at 8k against a 2% margin. A 2x2 ablation separating the control token from output restriction finds that the token changes performance by +0.07% to +0.13%, with all intervals crossing zero, while output restriction costs +0.47% to +1.19%; the factors are substitutes rather than complements. Multi-cap training nevertheless improves robustness: under typographical noise, the same checkpoint degrades 12.5--15.4 points less in its fine mode and outperforms each fixed-cap specialist at that specialist's vocabulary size. A control with neither cap token nor output restriction is equally robust, attributing this benefit to multi-granularity training rather than conditioning. The per-cap penalty tracks each cap's share of training rows, yielding a falsifiable prediction for future work.

Related papers

• Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models [67.72077328700088]
  We present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously.<n>We find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau.<n>By operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-kruncation during logit dumping.
  arXiv  Detail & Related papers  (2026-09-11T00:17:22Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Vowel Signs Are Not Letters: A Pre-tokenization Ceiling on Multilingual Tokenizer Fertility [0.0]
  In abugida scripts, vowels are written as combining marks; this pattern therefore splits each word at every vowel sign.<n>Since BPE merges only within a pre-token, those splits persist through training regardless of vocabulary size or corpus composition.<n>Across 26 languages from a parallel corpus, every one of the 17 abugidas is affected, ranging from 1.47x (Tibetan) to 9.02x (Thai)<n>For 5 languages, matched tokenizer pairs that differ only in this character class fall within 2.2% of the predicted floor, scoring 4.78 versus 1.58 tokens per word on Nepali.
  arXiv  Detail & Related papers  (2026-08-26T22:56:24Z)
• Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding [0.0]
  Pruned BPE is a post-training visibility-pruning and token-reallocation method.<n>It separates merge construction from model-visible vocabulary selection.<n>Experiments show that Pruned BPE consistently reduces encoded length relative to Standard BPE.
  arXiv  Detail & Related papers  (2026-08-01T19:34:44Z)
• In-Place Tokenizer Expansion for Pre-trained LLMs [47.56305202796458]
  We present an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design.<n>A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality.<n>The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4times$ and $2.6times$ fewer tokens than the source.
  arXiv  Detail & Related papers  (2026-07-16T17:32:38Z)
• Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls [0.0]
  Penalty is not well-defined.<n>It corrupts structured output.<n>On 200 real-world schemas, theta=1.3 drops the rate of valid, schema-conformant output from 97% to 23%.
  arXiv  Detail & Related papers  (2026-07-09T05:11:22Z)
• Bag of Dims: Training-Free Mechanistic Interpretability via Dimension-Level Sign Patterns [0.0]
  We show the standard basis of transformer hidden states already provides a training-free, architecture-general feature basis.<n>We validate this Bag of Dims framework across seven models spanning language.
  arXiv  Detail & Related papers  (2026-06-10T19:34:31Z)
• Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models [0.0]
  Large language models route every input through a learned embedding table of shape.<n>We introduce Kronecker Embeddings, a deterministic byte-level character-position factorization.<n>It eliminates 91--94% of input-side trainable parameters at frontier scale.
  arXiv  Detail & Related papers  (2026-05-28T06:53:18Z)
• Compute Optimal Tokenization [79.3815358070537]
  We investigate how the information granularity of tokens, controlled by the compression rate, affects scaling trends.<n>We train 988 latent tokenized models (BLT) ranging from 50M to 7B parameters that enable setting the desired compression rate.<n>Our experiments reveal that in compute-optimal configurations, model parameter counts scale proportionally to data size measured in bytes, not in tokens as commonly perceived.
  arXiv  Detail & Related papers  (2026-05-02T01:53:22Z)
• Compact Constraint Encoding for LLM Code Generation: An Empirical Study of Token Economics and Constraint Compliance [0.0]
  We investigate whether compact, structured constraint headers can reduce prompt token consumption without degrading constraint compliance.<n>We find that compact headers reduce constraint-portion tokens by approximately 71% and full-prompt tokens by 25-30%, replicated across three independent rounds.
  arXiv  Detail & Related papers  (2026-04-08T15:18:33Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Beyond Next Token Prediction: Patch-Level Training for Large Language Models [69.67438563485887]
  We introduce patch-level training for Large Language Models (LLMs)<n>During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch.<n>We show that patch-level training can reduce the overall training costs to 0.5$times$, without compromising the model performance.
  arXiv  Detail & Related papers  (2024-07-17T15:48:39Z)
• Rho-1: Not All Tokens Are What You Need [132.31428897792114]
  Previous language model pre-training methods uniformly applied a next-token prediction loss to all training tokens.<n>Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that align with desired distribution.<n>When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks.
  arXiv  Detail & Related papers  (2024-04-11T17:52:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.