Dynamic Multi-Byte Prediction With Hierarchical Language Models

• URL: http://arxiv.org/abs/2608.15454v2
• Date: Fri, 21 Aug 2026 23:00:48 GMT
• Title: Dynamic Multi-Byte Prediction With Hierarchical Language Models
• Abstract summary: We introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters.<n>First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM.<n>Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality.
• Score: 12.377460054545944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput.

Related papers

• DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models [2.3863052459868297]
  We propose a training-free decoding strategy that leverages inter-token dependencies to inform token updates during generation.<n> Empirical results demonstrate that DOS consistently achieves superior performance on both code generation and mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2026-03-16T14:28:39Z)
• Parallel Token Prediction for Language Models [29.843197713818885]
  We propose Parallel Token Prediction (PTP), a universal framework for parallel sequence generation in language models.<n>PTP jointly predicts multiple dependent tokens in a single transformer call by incorporating the sampling procedure into the model.<n>We prove that PTP can represent arbitrary autoregressive sequence distributions.
  arXiv  Detail & Related papers  (2025-12-24T18:46:55Z)
• Continuous Autoregressive Language Models [56.49239051750678]
  We introduce Continuous Autoregressive Language Models (CALM)<n>CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector.<n>We develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling.
  arXiv  Detail & Related papers  (2025-10-31T17:58:11Z)
• Frame-Stacked Local Transformers For Efficient Multi-Codebook Speech Generation [13.289870835946347]
  Speech generation models based on large language models (LLMs) operate on discrete acoustic codes, which differ fundamentally from text tokens due to their multicodebook structure.<n>At each timestep, models must predict N codebook entries jointly, introducing dependencies that challenge simple parallel prediction approaches.<n>To address this, hierarchical strategies employ a local transformer (LT) to refine predictions and capture intra-timestep dependencies.<n>We propose practical guidelines for selecting decoding strategies based on deployment priorities such as computational efficiency and synthesis fidelity.
  arXiv  Detail & Related papers  (2025-09-23T21:31:00Z)
• Sampling from Your Language Model One Byte at a Time [82.71473348639489]
  Tokenization can introduce distortion into the model's generations, known as the Prompt Boundary Problem (PBP)<n>We present an inference-time method to convert any autore LM with a BPE tokenizer into a character-level or byte-level LM.<n>Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers.
  arXiv  Detail & Related papers  (2025-06-17T02:37:04Z)
• L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models [95.53699156138435]
  We propose leap multi-token prediction(L-MTP), an innovative token prediction method.<n>Unlike conventional MTP, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass.<n>We theoretically demonstrate the benefit of L-MTP in improving inference efficiency.
  arXiv  Detail & Related papers  (2025-05-23T05:59:46Z)
• FIRP: Faster LLM inference via future intermediate representation prediction [54.897493351694195]
  FIRP generates multiple tokens instead of one at each decoding step. We conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets.
  arXiv  Detail & Related papers  (2024-10-27T15:53:49Z)
• Graph-Structured Speculative Decoding [52.94367724136063]
  Speculative decoding has emerged as a promising technique to accelerate the inference of Large Language Models. We introduce an innovative approach utilizing a directed acyclic graph (DAG) to manage the drafted hypotheses. We observe a remarkable speedup of 1.73$times$ to 1.96$times$, significantly surpassing standard speculative decoding.
  arXiv  Detail & Related papers  (2024-07-23T06:21:24Z)
• Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration [54.897493351694195]
  We propose a novel parallel decoding approach, namely textithidden transfer, which decodes multiple successive tokens simultaneously in a single forward pass. In terms of acceleration metrics, we outperform all the single-model acceleration techniques, including Medusa and Self-Speculative decoding.
  arXiv  Detail & Related papers  (2024-04-18T09:17:06Z)
• SPEED: Speculative Pipelined Execution for Efficient Decoding [35.45955948053644]
  We propose SPEED, which improves inference efficiency by speculatively executing multiple future tokens in parallel with the current token. For Transformer decoders that employ parameter sharing, the memory operations for the tokens executing in parallel can be amortized. We demonstrate the efficiency of our method in terms of latency reduction relative to model accuracy and demonstrate how speculation allows for training deeper decoders with parameter sharing with minimal runtime overhead.
  arXiv  Detail & Related papers  (2023-10-18T16:07:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.