P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling

• URL: http://arxiv.org/abs/2606.24447v1
• Date: Tue, 23 Jun 2026 11:34:28 GMT
• Title: P-MTP: Efficient Document Parsing via Multi-Token Prediction with Progressive Depth Scaling
• Abstract summary: Vision-Language Models (VLMs) have revolutionized document parsing by enabling end-to-end mapping from images to structured text.<n>While Multi-Token Prediction (MTP) has emerged as a promising approach for accelerating inference, its potential is constrained by optimization instability when scaling to deeper look-ahead depth.<n>We propose textbfP-MTP, a framework that leverages textbf Multi-Token Prediction with a lightweight MTP module to scale the look-ahead depth.
• Score: 9.276742450240961
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have revolutionized document parsing by enabling end-to-end mapping from images to structured text, imposing a significant latency bottleneck, particularly for token-dense documents. While Multi-Token Prediction (MTP) has emerged as a promising approach for accelerating inference, its potential is constrained by optimization instability when scaling to deeper look-ahead depth. In this paper, we propose \textbf{P-MTP}, a framework that leverages \textbf{Progressive Multi-Token Prediction} with a lightweight MTP module to scale the look-ahead depth for high-throughput document parsing. Specifically, we introduce Progressive Curriculum Loss that adaptively re-weights different look-ahead depths using cumulative path reliability and retrospective target consistency. By effectively suppressing gradient noise in long-range predictions, P-MTP, facilitates an automated easy-to-hard optimization transition, enabling the model to master increasingly distant look-ahead depths. Furthermore, we propose Confidence-Gated Dynamic Drafting to maximize the effective look-ahead depth and acceptance rate by adaptively calibrating speculative length during inference, thereby minimizing computational waste and further pushing the boundaries of inference speedup. Experimental results across multiple benchmarks and architectures demonstrate that P-MTP, achieves up to a $5\times$ speedup with negligible loss in accuracy, providing the first successful validation of extensive look-ahead MTP in the document parsing domain.

Related papers

• AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction [32.62594861280837]
  Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads.<n>We propose AdaMTP, an adaptive training paradigm that aligns the prediction horizon with the intrinsic predictability of the sequence.<n>Across mathematical reasoning, code generation, and general benchmarks on three backbones, AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.
  arXiv  Detail & Related papers  (2026-08-01T04:18:49Z)
• How Transformers Learn to Plan via Multi-Token Prediction [33.974020982252206]
  Multi-token prediction (MTP) has emerged as a promising alternative to next-token prediction (NTP)<n>We show that MTP consistently outperforms NTP on both synthetic graph path-finding tasks and more realistic reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-04-13T18:04:09Z)
• Self-Distillation for Multi-Token Prediction [27.12687724768597]
  Multi-Token Prediction (MTP) could accelerate inference by predicting multiple future tokens in parallel.<n>We propose MTP-D, a simple yet effective self-distillation method with minimal additional training cost.<n>We also introduce a looped extension strategy for MTP-D, enabling effective and economical MTP head extension.
  arXiv  Detail & Related papers  (2026-03-25T04:00:29Z)
• Training-Free Acceleration for Document Parsing Vision-Language Model with Hierarchical Speculative Decoding [102.88996030431662]
  We propose a training-free and highly efficient acceleration method for document parsing tasks.<n>Inspired by speculative decoding, we employ a lightweight document parsing pipeline as a draft model to predict batches of future tokens.<n>We demonstrate the effectiveness of our approach on the general-purpose OmniDocBench.
  arXiv  Detail & Related papers  (2026-02-13T14:22:10Z)
• Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings [52.49524240846879]
  We propose Hierarchical Token Prepending to mitigate attention-level compression and readout-level over-squashing.<n>HTP partitions the input into blocks and prepends block-level summary tokens to subsequent blocks, creating pathways for backward information flow.<n>As a simple, architecture-agnostic method, HTP enhances both zero-shot and finetuned models, offering a scalable route to superior long-document embeddings.
  arXiv  Detail & Related papers  (2025-11-18T19:37:40Z)
• MTP-S2UT: Enhancing Speech-to-Speech Translation Quality with Multi-token Prediction [49.92201266421949]
  We introduce multi-token prediction (MTP) loss into speech-to-unit translation (S2UT) models.<n>We show that all MTP loss variants consistently improve the quality of S2UT translation.
  arXiv  Detail & Related papers  (2025-10-11T04:06:20Z)
• FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction [11.691960175716163]
  This paper introduces FastMTP, a method that improves multi-step draft quality by aligning MTP training with its inference pattern.<n>Our approach fine-tunes a single MTP head with position-shared weights on self-distilled data, enabling it to capture dependencies among consecutive future tokens.<n> Experimental results across seven diverse benchmarks demonstrate that FastMTP achieves an average of 2.03x speedup compared to standard next token prediction.
  arXiv  Detail & Related papers  (2025-09-16T07:36:26Z)
• Pre-Training Curriculum for Multi-Token Prediction in Language Models [2.8071268036220003]
  Multi-token prediction (MTP) is a recently proposed pre-training objective for language models.<n>We propose a curriculum learning strategy for MTP training, exploring two variants: a forward curriculum and a reverse curriculum.
  arXiv  Detail & Related papers  (2025-05-28T18:19:18Z)
• L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models [95.53699156138435]
  We propose leap multi-token prediction(L-MTP), an innovative token prediction method.<n>Unlike conventional MTP, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass.<n>We theoretically demonstrate the benefit of L-MTP in improving inference efficiency.
  arXiv  Detail & Related papers  (2025-05-23T05:59:46Z)
• Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference [54.53508601749513]
  We propose a task-oriented feature compression (TOFC) method for multimodal understanding in a device-edge co-inference framework.<n>To enhance compression efficiency, multiple entropy models are adaptively selected based on the characteristics of the visual features.<n>Results show that TOFC achieves up to 52% reduction in data transmission overhead and 63% reduction in system latency.
  arXiv  Detail & Related papers  (2025-03-17T08:37:22Z)
• MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer [66.71930982549028]
  Vision-Language Transformers (VLTs) have shown great success recently, but are accompanied by heavy computation costs. We propose a novel framework named Multimodal Alignment-Guided Dynamic Token Pruning (MADTP) for accelerating various VLTs.
  arXiv  Detail & Related papers  (2024-03-05T14:13:50Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.