Fast Byte Latent Transformer

• URL: http://arxiv.org/abs/2605.08044v1
• Date: Fri, 08 May 2026 17:35:27 GMT
• Title: Fast Byte Latent Transformer
• Abstract summary: We introduce BLT Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise diffusion objective alongside the standard next-byte prediction loss.<n>Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality.<n>All methods may achieve an estimated memory-bandwidth cost over 50% lower than BLT on generation tasks.
• Score: 73.03308456251764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent byte-level language models (LMs) match the performance of token-level models without relying on subword vocabularies, yet their utility is limited by slow, byte-by-byte autoregressive generation. We address this bottleneck in the Byte Latent Transformer (BLT) through new training and generation techniques. First, we introduce BLT Diffusion (BLT-D), a new model and our fastest BLT variant, trained with an auxiliary block-wise diffusion objective alongside the standard next-byte prediction loss. This enables an inference procedure that generates multiple bytes in parallel per decoding step, substantially reducing the number of forward passes required to generate a sequence. Second, we propose two extensions inspired by speculative decoding that trade some of this speed for higher generation quality: BLT Self-speculation (BLT-S), in which BLT's local decoder continues generating past its normal patch boundaries to draft bytes, which are then verified with a single full-model forward pass; and BLT Diffusion+Verification (BLT-DV), which augments BLT-D with an autoregressive verification step after diffusion-based generation. All methods may achieve an estimated memory-bandwidth cost over 50% lower than BLT on generation tasks. Each approach offers its own unique advantages, together removing key barriers to the practical use of byte-level LMs.

Related papers

• Cross-Tokenizer LLM Distillation through a Byte-Level Interface [47.989175659742294]
  Cross-tokenizer distillation (CTD) is the transfer of knowledge from a teacher to a student language model when the two use different tokenizers.<n>We propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level.
  arXiv  Detail & Related papers  (2026-04-08T18:05:38Z)
• Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLM [58.322826487307765]
  We present Fast-dVLM, a block-diffusion-based VLM that enables KV-cache-compatible parallel decoding and speculative block decoding for inference acceleration.<n>With SGLang integration and FP8 quantization, Fast-dVLM achieves over 6x end-to-end inference speedup over the AR baseline.
  arXiv  Detail & Related papers  (2026-04-08T08:50:08Z)
• dParallel: Learnable Parallel Decoding for dLLMs [77.24184219948337]
  Diffusion large language models (dLLMs) offer parallel token prediction and lower inference latency.<n>Existing open-source models still require nearly token-length decoding steps to ensure performance.<n>We introduce dParallel, a simple and effective method that unlocks the inherent parallelism of dLLMs for fast sampling.
  arXiv  Detail & Related papers  (2025-09-30T16:32:52Z)
• Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding [10.223767541511991]
  SubSpec is a plug-and-play method to accelerate parameter offloading.<n>It achieves 9.1x speedup for Qwen2.5 7B on MT-Bench and 12.5x speedup for Qwen2.5 32B on popular generation benchmarks.
  arXiv  Detail & Related papers  (2025-09-22T19:08:57Z)
• Accelerating Diffusion LLMs via Adaptive Parallel Decoding [60.407727995313074]
  We introduce adaptive parallel decoding (APD), a novel method that dynamically adjusts the number of tokens sampled in parallel.<n>APD provides markedly higher throughput with minimal quality degradations on downstream benchmarks.
  arXiv  Detail & Related papers  (2025-05-31T06:10:10Z)
• Byte Latent Transformer: Patches Scale Better Than Tokens [101.10994909832063]
  Byte Latent Transformer (BLT) encodes bytes into dynamically sized patches, which serve as the primary units of computation.<n>For fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.
  arXiv  Detail & Related papers  (2024-12-13T05:33:32Z)
• BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models [37.09385961422664]
  Large language models (LLMs) commonly employ autoregressive generation during inference, leading to high memory bandwidth demand and extended latency. We present Bi-directional Tuning for lossless Acceleration (BiTA), an innovative method expediting LLMs via streamlined semi-autoregressive generation and draft verification. The proposed BiTA, LLaMA-2-70B-Chat achieves a 2.7$times$ speedup on the MT-Bench benchmark.
  arXiv  Detail & Related papers  (2024-01-23T06:36:49Z)
• Blockwise Parallel Transformer for Large Context Models [70.97386897478238]
  Blockwise Parallel Transformer (BPT) is a blockwise computation of self-attention and feedforward network fusion to minimize memory costs. By processing longer input sequences while maintaining memory efficiency, BPT enables training sequences 32 times longer than vanilla Transformers and up to 4 times longer than previous memory-efficient methods.
  arXiv  Detail & Related papers  (2023-05-30T19:25:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.