Hierarchical Latent Prediction for Language Models

• URL: http://arxiv.org/abs/2608.05806v1
• Date: Thu, 06 Aug 2026 09:41:41 GMT
• Title: Hierarchical Latent Prediction for Language Models
• Abstract summary: Hierarchical Latent Prediction (HiLP) is an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts.<n>HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks.
• Score: 14.925286273360598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.

Related papers

• Learning with Foresight: Enhancing Neural Routing Policy via Multi-Node Lookahead Prediction [53.32115440294569]
  Multi-node Lookahead Prediction (MnLP) is a novel training strategy that extends the supervised learning paradigm to predict multiple future nodes simultaneously.<n>We incorporate myopic causal and discardable MnLP modules that operate exclusively during training, facilitating contextual models to anticipate multi-step decisions.<n> Experimentally, MnLP outperforms existing training methods, improving the generalization capability of neural policies across various problem sizes, distributions, and real-world benchmarks.
  arXiv  Detail & Related papers  (2026-05-19T15:19:59Z)
• Possibilistic Predictive Uncertainty for Deep Learning [75.00607359322863]
  We introduce Dirichlet-approximated possibilistic posterior predictions (DAPPr)<n>DAPPr is a principled framework leveraging possibility theory.<n>This projection-and-approximation strategy yields a simple training objective with closed-form solutions.
  arXiv  Detail & Related papers  (2026-05-01T12:14:01Z)
• Latent Chain-of-Thought for Visual Reasoning [53.541579327424046]
  Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs)<n>We reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference.<n>We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-27T23:10:06Z)
• Context-level Language Modeling by Learning Predictive Context Embeddings [79.00607069677393]
  We introduce textbfContextLM, a framework that augments standard pretraining with an inherent textbfnext-context prediction objective.<n>This mechanism trains the model to learn predictive representations of multi-token contexts, leveraging error signals derived from future token chunks.<n>Experiments on the GPT2 and Pythia model families, scaled up to $1.5$B parameters, show that ContextLM delivers consistent improvements in both perplexity and downstream task performance.
  arXiv  Detail & Related papers  (2025-10-23T07:09:45Z)
• Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach [24.85612231267623]
  Traditional reinforcement learning assumes the agents make decisions based on Markov decision processes (MDPs) with one-step transition models.<n>In many real-world applications, such as energy management and stock investment, agents can access multi-step predictions of future states.<n>We introduce BOLA, a two-stage model-based RL algorithm that separates offline Bayesian value learning from lightweight online adaptation to real-time predictions.
  arXiv  Detail & Related papers  (2025-10-21T14:47:08Z)
• Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries [35.39150917025755]
  Future summary prediction (FSP) trains an auxiliary head to predict a compact representation of the long-term future.<n>FSP provides improvements over both NTP and MTP across math, reasoning, and coding benchmarks.
  arXiv  Detail & Related papers  (2025-10-16T14:52:52Z)
• Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective [12.655285605773932]
  We show that Transformers indeed struggle with high-ambiguity predictions across model sizes.<n>Preliminary results show substantial gains in ambiguous contexts through improved capacity allocation and test-time scalable inference.
  arXiv  Detail & Related papers  (2025-06-19T13:05:12Z)
• L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models [95.53699156138435]
  We propose leap multi-token prediction(L-MTP), an innovative token prediction method.<n>Unlike conventional MTP, L-MTP strategically skips over intermediate tokens, predicting non-sequential ones in a single forward pass.<n>We theoretically demonstrate the benefit of L-MTP in improving inference efficiency.
  arXiv  Detail & Related papers  (2025-05-23T05:59:46Z)
• Efficient Joint Prediction of Multiple Future Tokens [20.647830092055955]
  We introduce joint multi-token prediction (JTP), a lightweight modification of standard next-token prediction.<n>Unlike previous multi-token prediction approaches, JTP strategically employs teacher forcing of future-tokens.<n>We show that the JTP approach achieves a short-horizon belief state representation, while popular alternatives for multi-token prediction fail to do so.
  arXiv  Detail & Related papers  (2025-03-24T19:52:42Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• Aligned Contrastive Predictive Coding [10.521845940927163]
  We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned.
  arXiv  Detail & Related papers  (2021-04-24T13:07:22Z)
• Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
  We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
  arXiv  Detail & Related papers  (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.