MARS: Enabling Autoregressive Models Multi-Token Generation

• URL: http://arxiv.org/abs/2604.07023v1
• Date: Wed, 08 Apr 2026 12:41:37 GMT
• Title: MARS: Enabling Autoregressive Models Multi-Token Generation
• Abstract summary: Autoregressive (AR) language models generate text one token at a time, even when consecutive tokens are highly predictable given earlier context.<n>We introduce MARS, a lightweight fine-tuning method that teaches an instruction-tuned AR model to predict multiple tokens per forward pass.
• Score: 34.63470140050534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoregressive (AR) language models generate text one token at a time, even when consecutive tokens are highly predictable given earlier context. We introduce MARS (Mask AutoRegreSsion), a lightweight fine-tuning method that teaches an instruction-tuned AR model to predict multiple tokens per forward pass. MARS adds no architectural modifications, no extra parameters, and produces a single model that can still be called exactly like the original AR model with no performance degradation. Unlike speculative decoding, which maintains a separate draft model alongside the target, or multi-head approaches such as Medusa, which attach additional prediction heads, MARS requires only continued training on existing instruction data. When generating one token per forward pass, MARS matches or exceeds the AR baseline on six standard benchmarks. When allowed to accept multiple tokens per step, it maintains baseline-level accuracy while achieving 1.5-1.7x throughput. We further develop a block-level KV caching strategy for batch inference, achieving up to 1.71x wall-clock speedup over AR with KV cache on Qwen2.5-7B. Finally, MARS supports real-time speed adjustment via confidence thresholding: under high request load, the serving system can increase throughput on the fly without swapping models or restarting, providing a practical latency-quality knob for deployment.

Related papers

• K-Forcing: Joint Next-K-Token Decoding via Push-Forward Language Modeling [37.46942663162738]
  K-Forcing is a push-forward language modeling paradigm for joint next-k-token decoding.<n>We evaluate K-Forcing on LM1B and OpenWebText using a standard causal Transformer backbone.
  arXiv  Detail & Related papers  (2026-06-09T13:02:00Z)
• Multi-Token Residual Prediction [21.234596379356724]
  Diffusion Language Models generate text by iteratively denoising masked token sequences.<n>We introduce Multi-token Residual Prediction (MRP), a lightweight module that enables dependency-aware multi-token denoising.<n>MRP predicts the residual between steps from the backbone's hidden states, effectively denoising more tokens per backbone forward at a fraction of the cost.
  arXiv  Detail & Related papers  (2026-05-12T11:40:39Z)
• Switch-Hurdle: A MoE Encoder with AR Hurdle Decoder for Intermittent Demand Forecasting [0.0]
  Intermittent demand is a persistent challenge in retail and supply chain forecasting.<n>We introduce Switch-Hurdle: a new framework that integrates a Mixture-of-Experts (MoE) encoder with a Hurdle-based probabilistic decoder.
  arXiv  Detail & Related papers  (2026-02-26T07:03:19Z)
• WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference [44.87788417755154]
  We propose WeDLM, a diffusion decoding framework built entirely on standard causal attention.<n>We show that WeDLM preserves the quality of strong AR backbones while delivering substantial speedups.
  arXiv  Detail & Related papers  (2025-12-28T01:25:48Z)
• TiDAR: Think in Diffusion, Talk in Autoregression [59.94106070312094]
  TiDAR is a sequence-level hybrid architecture that drafts tokens (Thinking) in Diffusion and samples final outputs (Talking) AutoRegressively.<n> TiDAR is the first architecture to close the quality gap with AR models while delivering 4.71x to 5.91x more tokens per second.
  arXiv  Detail & Related papers  (2025-11-12T02:59:33Z)
• Diffusion Language Models are Super Data Learners [61.721441061210896]
  When unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs.<n>We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation.
  arXiv  Detail & Related papers  (2025-11-05T08:17:42Z)
• REAR: Rethinking Visual Autoregressive Models via Generator-Tokenizer Consistency Regularization [130.46612643194973]
  reAR is a simple training strategy introducing a token-wise regularization objective.<n>On ImageNet, it reduces gFID from 3.02 to 1.86 and improves IS to 316.9 using a standardization-based tokenizer.<n>When applied to advanced tokenizers, it achieves a gFID of 1.42 with only 177M parameters, matching the performance with larger state-of-the-art diffusion models (675M)
  arXiv  Detail & Related papers  (2025-10-06T02:48:13Z)
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding [55.2480439325792]
  In arbitrary-order language models, it is an open question how to sample tokens in parallel from the correct joint distribution.<n>We find that a different class of models, any-subset autoregressive models (AS-ARMs), holds the solution.<n>We show that AS-ARMs achieve state-of-the-art performance among sub-200M parameter models on infilling benchmark tasks, and nearly match the performance of models 50X larger on code generation.
  arXiv  Detail & Related papers  (2025-04-29T06:33:13Z)
• LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching [33.024044212891326]
  Masked Autoregressive (MAR) models have emerged as a promising approach in image generation.<n>We propose LazyMAR, which introduces two caching mechanisms to handle them one by one.<n>Our method achieves 2.83 times acceleration with almost no drop in generation quality.
  arXiv  Detail & Related papers  (2025-03-16T10:54:59Z)
• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition [62.83832841523525]
  We propose a fast and accurate parallel transformer, termed Paraformer. It accurately predicts the number of output tokens and extract hidden variables. It can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
  arXiv  Detail & Related papers  (2022-06-16T17:24:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.