OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

• URL: http://arxiv.org/abs/2609.01068v2
• Date: Thu, 10 Sep 2026 08:40:16 GMT
• Title: OUTLETS: Output-Length Prediction from Speculative Decoding Backbones
• Abstract summary: Heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling.<n>We identify a structural connection between speculative decoding (SD) and length prediction.<n>We introduce OUTLETS, which repurposes the speculative backbone as a trajectory-aware length predictor.
• Score: 9.273938403693576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.

Related papers

• ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping [79.1905186547363]
  ResiSpec is a framework that strategically reforms the proposal distribution during verification.<n>It prevents candidate obsolescence and achieves up to 1.92$times$ speedup over state-of-the-art multi-candidate methods.
  arXiv  Detail & Related papers  (2026-08-25T11:25:55Z)
• Predict-then-Diffuse: Adaptive Response Length for Compute-Budgeted Inference in Diffusion LLMs [7.630708097054872]
  Diffusion-based Large Language Models (D-LLMs) represent a promising frontier in generative AI.<n>We propose Predict-then-Diffuse, a model-agnostic framework that enables compute-budgeted inference per input query.<n>We show that Predict-then-Diffuse significantly reduces computational costs (FLOP) compared to the default D-LLM inference mechanism.
  arXiv  Detail & Related papers  (2026-05-05T18:55:24Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions [16.877407702260243]
  Existing methods usually predict a single output length for each request to facilitate scheduling.<n>We propose Tail Inflated Expectation (TIE) to replace the output length in SJF scheduling.<n>TIE reduces the per-token latency by $2.31times$ for online inference and improves throughput by $1.42times for offline data generation.
  arXiv  Detail & Related papers  (2026-04-01T05:31:21Z)
• Predicting LLM Output Length via Entropy-Guided Representations [13.351384070796747]
  We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction.<n>Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction.
  arXiv  Detail & Related papers  (2026-02-12T10:49:04Z)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios [76.85739138203014]
  We present SpecFormer, a novel architecture that accelerates unidirectional and attention mechanisms.<n>We demonstrate that SpecFormer achieves lower training demands and reduced computational costs.
  arXiv  Detail & Related papers  (2025-11-25T14:20:08Z)
• Fast Inference via Hierarchical Speculative Decoding [65.40448210801763]
  We introduce Hierarchical Speculative Decoding (HSD), an algorithm that stacks draft models into a hierarchy, where each model proposes tokens, and the next larger model verifies them in a single forward pass.<n>HSD gives up to 1.2x speed-up over the best single-draft baseline.
  arXiv  Detail & Related papers  (2025-10-22T15:56:19Z)
• SpecExit: Accelerating Large Reasoning Model via Speculative Exit [10.522333173441453]
  We propose SpecExit, a framework that predicts both future tokens and an early-exit signal directly from a draft model without probing overhead.<n>Our method offers significant improvements, reducing average generation length by 66% and achieving a 2.5x speedup in end-to-end latency.
  arXiv  Detail & Related papers  (2025-09-29T03:39:32Z)
• $\texttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts [55.231201692232894]
  $textttSPECS$ is a latency-aware test-time scaling method inspired by speculative decoding.<n>Our results show that $textttSPECS$matches or surpasses beam search accuracy while reducing latency by up to $sim$19.1%.
  arXiv  Detail & Related papers  (2025-06-15T05:50:05Z)
• Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation [64.59292053188264]
  Experimental results on mainstream SD benchmarks as well as reasoning-heavy benchmarks demonstrate the superior performance of SVIP.<n>SVIP is a training-free dynamic length policy for speculative decoding systems that adaptively determines the lengths of draft sequences by referring to the draft entropy.
  arXiv  Detail & Related papers  (2024-11-27T15:53:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.