The pitfalls of next-token prediction

• URL: http://arxiv.org/abs/2403.06963v3
• Date: Tue, 29 Jul 2025 03:57:01 GMT
• Title: The pitfalls of next-token prediction
• Abstract summary: We argue that the two often-conflated phases of next-token prediction must be treated distinctly.<n>The popular criticism that errors can compound during autoregressive inference, crucially assumes that teacher-forcing has learned an accurate next-token predictor.<n>We describe a general mechanism of how teacher-forcing can fail, and design a minimal planning task where both the Transformer and the Mamba architecture empirically fail.
• Score: 18.68202374460919
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Can a mere next-token predictor faithfully model human intelligence? We crystallize this emerging concern and correct popular misconceptions surrounding it, and advocate a simple multi-token objective. As a starting point, we argue that the two often-conflated phases of next-token prediction -- autoregressive inference and teacher-forced training -- must be treated distinctly. The popular criticism that errors can compound during autoregressive inference, crucially assumes that teacher-forcing has learned an accurate next-token predictor. This assumption sidesteps a more deep-rooted problem we expose: in certain classes of tasks, teacher-forcing can simply fail to learn an accurate next-token predictor in the first place. We describe a general mechanism of how teacher-forcing can fail, and design a minimal planning task where both the Transformer and the Mamba architecture empirically fail in that manner -- remarkably, despite the task being straightforward to learn. Finally, we provide preliminary evidence that this failure can be resolved using _teacherless_ training, a simple modification using dummy tokens that predicts multiple tokens in advance. We hope this finding can ground future debates and inspire explorations beyond the next-token prediction paradigm. We make our code available under https://github.com/gregorbachmann/Next-Token-Failures

Related papers

• Thinking into the Future: Latent Lookahead Training for Transformers [34.73973224120233]
  Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time.<n>We introduce latent lookahead, a training strategy that enables models to "think" before generating.<n>We show that latent lookahead substantially outperforms both autoregressive and non-autoregressive baselines on planning tasks such as maze solving, Sudoku, and ProsQA.
  arXiv  Detail & Related papers  (2026-03-03T17:15:42Z)
• Are you going to finish that? A Practical Study of the Partial Token Problem [85.49816027251013]
  Language models (LMs) are trained over sequences of tokens, whereas users interact with LMs via text.<n>This mismatch gives rise to the partial token problem, which occurs when a user ends their prompt in the middle of the expected next-token.<n>In this work, we identify three domains where token and "word" boundaries often do not line up.
  arXiv  Detail & Related papers  (2026-01-30T17:47:16Z)
• Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential [12.719829360337833]
  We propose a novel framework that leverages the inherent knowledge of vanilla autoregressive language models about future tokens.<n>Our method achieves significant speedups through supervised fine-tuning on pretrained models.
  arXiv  Detail & Related papers  (2025-07-16T02:31:40Z)
• Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective [12.655285605773932]
  We show that Transformers indeed struggle with high-ambiguity predictions across model sizes.<n>Preliminary results show substantial gains in ambiguous contexts through improved capacity allocation and test-time scalable inference.
  arXiv  Detail & Related papers  (2025-06-19T13:05:12Z)
• Improving Large Language Models with Concept-Aware Fine-Tuning [55.59287380665864]
  Concept-Aware Fine-Tuning (CAFT) is a novel multi-token training method for large language models (LLMs)<n>CAFT enables the learning of sequences that span multiple tokens, fostering stronger concept-aware learning.<n>Experiments demonstrate significant improvements compared to conventional next-token finetuning methods.
  arXiv  Detail & Related papers  (2025-06-09T14:55:00Z)
• Efficient Joint Prediction of Multiple Future Tokens [20.647830092055955]
  We introduce joint multi-token prediction (JTP), a lightweight modification of standard next-token prediction. Unlike previous multi-token prediction approaches, JTP strategically employs teacher forcing of future-tokens. We show that the JTP approach achieves a short-horizon belief state representation, while popular alternatives for multi-token prediction fail to do so.
  arXiv  Detail & Related papers  (2025-03-24T19:52:42Z)
• Measuring In-Context Computation Complexity via Hidden State Prediction [33.504027525492056]
  We show that a neural sequence model's ability to predict its own future hidden states correlates with the intuitive interestingness of the task. We propose a novel learned predictive prior that enables us to measure the novel information gained in each step. Our metric predicts the description length of formal languages learned in-context, the complexity of mathematical reasoning problems, and the correctness of self-generated reasoning chains.
  arXiv  Detail & Related papers  (2025-03-17T17:56:14Z)
• Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification [50.717692060500696]
  Next-token prediction with the logarithmic loss is a cornerstone of autoregressive sequence modeling. Next-token prediction can be made robust so as to achieve $C=tilde O(H)$, representing moderate error amplification. No computationally efficient algorithm can achieve sub-polynomial approximation factor $C=e(log H)1-Omega(1)$.
  arXiv  Detail & Related papers  (2025-02-18T02:52:00Z)
• Improving Next Tokens via Second-Last Predictions with Generate and Refine [1.8592384822257952]
  We train a decoder only architecture for predicting the second last token for a sequence of tokens. Our approach yields higher computational training efficiency than BERT-style models.
  arXiv  Detail & Related papers  (2024-11-23T22:09:58Z)
• FIRP: Faster LLM inference via future intermediate representation prediction [54.897493351694195]
  FIRP generates multiple tokens instead of one at each decoding step. We conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets.
  arXiv  Detail & Related papers  (2024-10-27T15:53:49Z)
• Semformer: Transformer Language Models with Semantic Planning [18.750863564495006]
  Next-token prediction serves as the dominant component in current neural language models. We introduce Semformer, a novel method of training a Transformer language model that explicitly models the semantic planning of response.
  arXiv  Detail & Related papers  (2024-09-17T12:54:34Z)
• TokenUnify: Scalable Autoregressive Visual Pre-training with Mixture Token Prediction [61.295716741720284]
  TokenUnify is a novel pretraining method that integrates random token prediction, next-token prediction, and next-all token prediction. Cooperated with TokenUnify, we have assembled a large-scale electron microscopy (EM) image dataset with ultra-high resolution. This dataset includes over 120 million annotated voxels, making it the largest neuron segmentation dataset to date.
  arXiv  Detail & Related papers  (2024-05-27T05:45:51Z)
• Better & Faster Large Language Models via Multi-token Prediction [29.067271500844928]
  Large language models such as GPT and Llama are trained with a next-token prediction loss. We suggest that training language models to predict multiple future tokens at once results in higher sample efficiency.
  arXiv  Detail & Related papers  (2024-04-30T17:33:57Z)
• TaskMet: Task-Driven Metric Learning for Model Learning [29.0053868393653]
  Deep learning models are often deployed in downstream tasks that the training procedure may not be aware of. We propose take the task loss signal one level deeper than the parameters of the model and use it to learn the parameters of the loss function the model is trained on. This approach does not alter the optimal prediction model itself, but rather changes the model learning to emphasize the information important for the downstream task.
  arXiv  Detail & Related papers  (2023-12-08T18:59:03Z)
• Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability [25.52470575274251]
  We observe that language models generate short repetitive phrases before learning to generate longer and more coherent text. Individual tokens often exhibit sudden increases or decreases in loss that are surprisingly consistent across pre-training runs. More frequent tokens reach lower final surprisals, exhibit less variability within and across pre-training runs, are learned earlier, and are less likely to be "forgotten" during pre-training.
  arXiv  Detail & Related papers  (2023-08-29T16:24:09Z)
• Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step [133.60124577507727]
  Chain-of-thought prompting primes large language models to verbalize rationalization for their predictions. We show that orders-of-magnitude smaller models can still benefit from chain-of-thought prompting. We introduce Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model.
  arXiv  Detail & Related papers  (2023-06-24T20:15:07Z)
• Uncertainty-Aware Unlikelihood Learning Improves Generative Aspect Sentiment Quad Prediction [52.05304897163256]
  We propose a template-agnostic method to control the token-level generation. Specifically, we introduce Monte Carlo dropout to understand the built-in uncertainty of pre-trained language models. We further propose marginalized unlikelihood learning to suppress the uncertainty-aware mistake tokens.
  arXiv  Detail & Related papers  (2023-06-01T07:49:06Z)
• Are socially-aware trajectory prediction models really socially-aware? [75.36961426916639]
  We introduce a socially-attended attack to assess the social understanding of prediction models. An attack is a small yet carefully-crafted perturbations to fail predictors. We show that our attack can be employed to increase the social understanding of state-of-the-art models.
  arXiv  Detail & Related papers  (2021-08-24T17:59:09Z)
• Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
  We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
  arXiv  Detail & Related papers  (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.