Cross-Tokenizer LLM Distillation through a Byte-Level Interface

• URL: http://arxiv.org/abs/2604.07466v2
• Date: Mon, 13 Apr 2026 14:15:52 GMT
• Title: Cross-Tokenizer LLM Distillation through a Byte-Level Interface
• Abstract summary: Cross-tokenizer distillation (CTD) is the transfer of knowledge from a teacher to a student language model when the two use different tokenizers.<n>We propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level.
• Score: 47.989175659742294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with--and on several benchmarks surpasses--significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.

Related papers

• Toppling the Hierarchy in Byte-level Language Modeling [54.626578706811436]
  State-of-the-art byte-level models use a hierarchical structure, starting at the byte level, downsampling to the word level, and then upsampling back to bytes.<n>We find that the hierarchical design itself limits character-level understanding, with pure byte-level models consistently outperforming hierarchical variants on character manipulation tasks.<n>Together, our results provide an explanation for the character-level failures of hierarchical byte models and establish a clear trade-off between computational efficiency and fine-grained character understanding.
  arXiv  Detail & Related papers  (2026-08-31T23:11:55Z)
• Fast Byte Latent Transformer [73.03308456251764]
  We introduce BLT Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise diffusion objective alongside the standard next-byte prediction loss.<n>Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality.<n>All methods may achieve an estimated memory-bandwidth cost over 50% lower than BLT on generation tasks.
  arXiv  Detail & Related papers  (2026-05-08T17:35:27Z)
• CKT-WAM: Parameter-Efficient Context Knowledge Transfer Between World Action Models [45.34156934993461]
  World action models (WAMs) provide a powerful generative framework for embodied control.<n>We propose textbfCKT-WAM, a framework that transfers teacher WAM's knowledge into a student WAM.<n>CKT-WAM consistently improves zero-shot generalization and achieves the best overall performance on LIBERO-Plus.
  arXiv  Detail & Related papers  (2026-05-07T13:26:51Z)
• Distilling Token-Trained Models into Byte-Level Models [16.418597104895245]
  Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization.<n>Existing BLMs typically require training from scratch on trillions of bytes, making them prohibitively expensive.<n>We propose an efficient distillation recipe that converts existing token-trained LLMs into BLMs while retaining comparable capabilities.
  arXiv  Detail & Related papers  (2026-02-01T04:16:49Z)
• Bolmo: Byteifying the Next Generation of Language Models [115.32940292418463]
  We introduce Bolmo, the first family of competitive fully open byte-level language models (LMs)<n>Byteification enables overcoming the limitations of subword tokenization.<n>We show that Bolmo can achieve inference speeds competitive with subword-level LMs.
  arXiv  Detail & Related papers  (2025-12-17T16:46:11Z)
• Learning what to say and how precisely: Efficient Communication via Differentiable Discrete Communication Learning [14.221641613518633]
  We develop a framework for end-to-end optimization of discrete messages.<n>We show it reduces bandwidth by over an order of magnitude while matching or exceeding task performance.
  arXiv  Detail & Related papers  (2025-11-03T13:16:57Z)
• Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models [81.74999702045339]
  Multi-Level Optimal Transport (MultiLevelOT) is a novel approach that advances the optimal transport for universal cross-tokenizer knowledge distillation.<n>Our method aligns the logit distributions of the teacher and the student at both token and sequence levels.<n>At the token level, MultiLevelOT integrates both global and local information by jointly optimizing all tokens within a sequence to enhance robustness.
  arXiv  Detail & Related papers  (2024-12-19T04:51:06Z)
• From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Control [58.72492647570062]
  We introduce our method -- Learnable Latent Codes as Bridges (LCB) -- as an alternate architecture to overcome limitations.<n>We find that methodoutperforms baselines that leverage pure language as the interface layer on tasks that require reasoning and multi-step behaviors.
  arXiv  Detail & Related papers  (2024-05-08T04:14:06Z)
• One Train for Two Tasks: An Encrypted Traffic Classification Framework Using Supervised Contrastive Learning [18.63871240173137]
  We propose an effective model named a Contrastive Learning Enhanced Temporal Fusion (CLE-TFE) In particular, we utilize supervised contrastive learning to enhance the packet-level and flow-level representations. We also propose cross-level multi-task learning, which simultaneously accomplishes the packet-level and flow-level classification tasks in the same model with one training.
  arXiv  Detail & Related papers  (2024-02-12T09:10:09Z)
• Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing [72.56219471145232]
  We propose a ST/MT multi-tasking framework with hard parameter sharing. Our method reduces the speech-text modality gap via a pre-processing stage. We show that our framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU.
  arXiv  Detail & Related papers  (2023-09-27T17:48:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.