Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions

• URL: http://arxiv.org/abs/2604.07931v1
• Date: Thu, 09 Apr 2026 07:49:52 GMT
• Title: Robust Length Prediction: A Perspective from Heavy-Tailed Prompt-Conditioned Distributions
• Abstract summary: Existing methods for output-length prediction are unreliable for prompt-only predictions.<n>We propose prompt-conditioned length distribution methods, which construct training targets from the same prompt.<n>Experiments across diverse scenarios show consistent gains in prediction quality.
• Score: 61.56973419225008
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Output-length prediction is important for efficient LLM serving, as it directly affects batching, memory reservation, and scheduling. For prompt-only length prediction, most existing methods use a one-shot sampled length as the label, implicitly treating each prompt as if it had one true target length. We show that this is unreliable: even under a fixed model and decoding setup, the same prompt induces a \emph{prompt-conditioned output length distribution}, not a deterministic scalar, and this distribution is consistent with \emph{heavy-tailed} behavior. Motivated by this, we cast length prediction as robust estimation from heavy-tailed prompt-conditioned length distributions. We propose prompt-conditioned length distribution (ProD) methods, which construct training targets from multiple independent generations of the same prompt. Two variants are developed to reuse the served LLM's hidden states: \mbox{ProD-M}, which uses a median-based target for robust point prediction, and ProD-D, which uses a distributional target that preserves prompt-conditioned uncertainty. We provide theoretical justifications by analyzing the estimation error under a surrogate model. Experiments across diverse scenarios show consistent gains in prediction quality.

Related papers

• ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping [79.1905186547363]
  ResiSpec is a framework that strategically reforms the proposal distribution during verification.<n>It prevents candidate obsolescence and achieves up to 1.92$times$ speedup over state-of-the-art multi-candidate methods.
  arXiv  Detail & Related papers  (2026-08-25T11:25:55Z)
• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• DBLAST: Dependent Block Drafting for Stochastic Speculative Decoding [26.72775347772003]
  We show that the accepted draft length degrades as the entropy of the target sampling distribution increases.<n>We propose a dependent block drafter based on a low-rank latent mixture over token positions, complemented by an acceptance-oriented training objective.
  arXiv  Detail & Related papers  (2026-08-05T22:49:54Z)
• PRESTO: Prefix-Aligned Tree Drafting for Diffusion Speculative Decoding [98.71600061506206]
  We propose PRESTO, a principled framework that extends tree-based drafting to diffusion drafters.<n> PRESTO achieves up to an average of $1.5times$ end-to-end throughput speedup on the state-of-the-art dedicated diffusion drafter SD.
  arXiv  Detail & Related papers  (2026-06-20T14:32:14Z)
• Asymptotically Log-Optimal Bayes-Assisted Confidence Sequences for Bounded Means [0.0]
  We propose a Bayes-assisted framework that uses a Bayesian working predictive model to adaptively construct confidence sequences.<n>Experiments on synthetic data, sequential best-arm identification for LLM evaluation, and prediction-powered inference show that informative priors can substantially reduce confidence-sequence width and sampling effort.
  arXiv  Detail & Related papers  (2026-05-08T16:27:50Z)
• Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions [16.877407702260243]
  Existing methods usually predict a single output length for each request to facilitate scheduling.<n>We propose Tail Inflated Expectation (TIE) to replace the output length in SJF scheduling.<n>TIE reduces the per-token latency by $2.31times$ for online inference and improves throughput by $1.42times for offline data generation.
  arXiv  Detail & Related papers  (2026-04-01T05:31:21Z)
• Adaptively Robust LLM Inference Optimization under Prediction Uncertainty [9.541681114575812]
  We study the problem of optimizing Large Language Model (LLM) inference scheduling to minimize total latency.<n>A key challenge in LLM inference scheduling is that while the prompt length is known upon arrival, the output length, which critically impacts memory usage and processing time, is unknown.<n>We propose algorithms that leverage machine learning to predict output lengths, assuming the prediction provides an interval classification (min-max range) for each request.
  arXiv  Detail & Related papers  (2025-08-20T08:55:26Z)
• Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation [64.59292053188264]
  Experimental results on mainstream SD benchmarks as well as reasoning-heavy benchmarks demonstrate the superior performance of SVIP.<n>SVIP is a training-free dynamic length policy for speculative decoding systems that adaptively determines the lengths of draft sequences by referring to the draft entropy.
  arXiv  Detail & Related papers  (2024-11-27T15:53:17Z)
• Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models [79.76293901420146]
  Large Language Models (LLMs) are employed across various high-stakes domains, where the reliability of their outputs is crucial. Our research investigates the fragility of uncertainty estimation and explores potential attacks. We demonstrate that an attacker can embed a backdoor in LLMs, which, when activated by a specific trigger in the input, manipulates the model's uncertainty without affecting the final output.
  arXiv  Detail & Related papers  (2024-07-15T23:41:11Z)
• DistPred: A Distribution-Free Probabilistic Inference Method for Regression and Forecasting [14.390842560217743]
  We propose a novel approach called DistPred for regression and forecasting tasks.<n>We transform proper scoring rules that measure the discrepancy between the predicted distribution and the target distribution into a differentiable discrete form.<n>This allows the model to sample numerous samples in a single forward pass to estimate the potential distribution of the response variable.
  arXiv  Detail & Related papers  (2024-06-17T10:33:00Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.