AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

• URL: http://arxiv.org/abs/2608.00434v1
• Date: Sat, 01 Aug 2026 04:18:49 GMT
• Title: AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
• Abstract summary: Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads.<n>We propose AdaMTP, an adaptive training paradigm that aligns the prediction horizon with the intrinsic predictability of the sequence.<n>Across mathematical reasoning, code generation, and general benchmarks on three backbones, AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.
• Score: 32.62594861280837
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision signal and accelerate inference. However, existing training frameworks adopt a rigid, fixed-length prediction horizon, disregarding the highly non-uniform information density of natural language and code. Forcing the auxiliary heads to predict across high-entropy semantic boundaries injects noisy, conflicting training signals; because these heads share the backbone's latent representations, the resulting gradients backpropagate and interfere with the model's core capabilities. We propose AdaMTP, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence. At its core, an entropy-based segmentation algorithm leverages the base model to detect sudden surges in uncertainty as semantic boundaries, partitioning sequences into variable-length groups. Each token is assigned an adaptive prediction depth, and a dynamically masked MTP objective suppresses the loss for predictions that cross these boundaries, attenuating the noisy gradients that degrade the backbone. Across mathematical reasoning, code generation, and general benchmarks on three backbones (Llama-3.1-8B, Qwen-2.5-7B, Gemma-3-12B), AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.

Related papers

• Predictive-Generative Drift Decomposition for Speech Enhancement and Separation [42.49632892186173]
  We propose a plug-and-play framework for speech enhancement and separation that augments predictive methods with a generative speech prior.<n>We decompose expressive dynamics into a task-specific drift and a denoising component, allowing a predictive estimate to be integrated directly into the generative sampling process.<n>This results in a mathematically grounded framework for combining strong pretrained predictors with the power of generative models.
  arXiv  Detail & Related papers  (2026-05-07T13:04:38Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendations [52.67948063133533]
  Generative Recommendation has emerged as a promising paradigm, reformulating recommendation as a sequence-to-sequence generation task over hierarchical Semantic IDs.<n>Existing methods suffer from a critical issue we term Semantic Drift, where errors in early, high-level tokens irreversibly divert the generation trajectory into irrelevant semantic subspaces.<n>We propose Promise, a novel framework that integrates dense, step-by-step verification into generative models.
  arXiv  Detail & Related papers  (2026-01-08T07:38:46Z)
• Infer As You Train: A Symmetric Paradigm of Masked Generative for Click-Through Rate Prediction [9.542597285477683]
  Generative models are increasingly being explored in click-through rate (CTR) prediction field.<n>Existing generative models typically confine the generative paradigm to the training phase.<n>We propose the Symmetric Masked Generative Paradigm for CTR prediction (SGCTR)<n>SGCTR applies the generative capabilities during online inference to iteratively mitigate the features of input samples.
  arXiv  Detail & Related papers  (2025-11-18T12:07:56Z)
• CaReTS: A Multi-Task Framework Unifying Classification and Regression for Time Series Forecasting [7.600702118055632]
  CaReTS is a novel multi-task learning framework that combines classification and regression tasks.<n>Four variants (CaReTS1--4) are instantiated under this framework to incorporate mainstream temporal modelling encoders.<n>Experiments on real-world datasets demonstrate that CaReTS outperforms state-of-the-art (SOTA) algorithms in forecasting accuracy.
  arXiv  Detail & Related papers  (2025-11-12T22:43:40Z)
• Next Interest Flow: A Generative Pre-training Paradigm for Recommender Systems by Modeling All-domain Movelines [8.895768051554162]
  We propose a novel generative pre-training paradigm for e-commerce recommender systems.<n>Our model learns to predict the Next Interest Flow, a dense vector sequence representing a user's future intent.<n>We present the All-domain Moveline Evolution Network (AMEN), a unified framework implementing our entire pipeline.
  arXiv  Detail & Related papers  (2025-10-13T12:13:17Z)
• Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective [12.655285605773932]
  We show that Transformers indeed struggle with high-ambiguity predictions across model sizes.<n>Preliminary results show substantial gains in ambiguous contexts through improved capacity allocation and test-time scalable inference.
  arXiv  Detail & Related papers  (2025-06-19T13:05:12Z)
• ALOcc: Adaptive Lifting-Based 3D Semantic Occupancy and Cost Volume-Based Flow Predictions [91.55655961014027]
  3D semantic occupancy and flow prediction are fundamental to understanding scene scene.<n>This paper proposes a vision-based framework with three targeted improvements.<n>Our purely convolutional architecture establishes new SOTA performance on multiple benchmarks for both semantic occupancy and joint semantic-flow prediction.
  arXiv  Detail & Related papers  (2024-11-12T11:32:56Z)
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion [61.03681839276652]
  Diffusion Forcing is a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels.<n>We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens.
  arXiv  Detail & Related papers  (2024-07-01T15:43:25Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)
• Exploiting Diffusion Prior for Generalizable Dense Prediction [85.4563592053464]
  Recent advanced Text-to-Image (T2I) diffusion models are sometimes too imaginative for existing off-the-shelf dense predictors to estimate. We introduce DMP, a pipeline utilizing pre-trained T2I models as a prior for dense prediction tasks. Despite limited-domain training data, the approach yields faithful estimations for arbitrary images, surpassing existing state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2023-11-30T18:59:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.