Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees

• URL: http://arxiv.org/abs/2608.30072v1
• Date: Sun, 30 Aug 2026 22:33:15 GMT
• Title: Learning Representations through Token Prediction: Geometry, Approximation, and Downstream Guarantees
• Abstract summary: Token prediction is a central pre-training objective for modern language models.<n>We develop a statistical framework connecting token prediction with representation geometry, encoder approximation, and downstream performance.
• Score: 6.750611816045116
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Token prediction is a central pre-training objective for modern language models. Despite its empirical success, why token prediction learns broadly useful representations remains incompletely understood. We develop a statistical framework connecting token prediction with representation geometry, encoder approximation, and downstream performance. Under a softmax prediction head, we show that accurate token prediction organizes token embeddings according to similarities between the distributions of contexts in which different token types appear, as measured by Hellinger distance, with explicit errors governed by prediction accuracy and token frequency. Meanwhile, the contextual representation provides a low-dimensional coordinate for the conditional distribution of the target token relative to these embeddings. We further introduce a self-consistency principle showing that repeated applications of a shared representation block can progressively refine the contextual representation without introducing additional block parameters. Among representations with the same prediction accuracy, this recurrent construction favors those that can be stably reconstructed from their contexts. Finally, we establish downstream guarantees for token generation, token community recovery, and classification by a linear probe, showing how prediction accuracy and recovered geometry translate into performance beyond the pre-training objective. Together, these results explain how the simple objective of predicting tokens can recover semantic geometry and produce broadly useful representations. A controlled simulation illustrates the theoretical mechanisms.

Related papers

• Grid-World Representations in Transformers Reflect Predictive Geometry [0.0]
  Next-token predictors often appear to develop internal representations of the latent world and its rules.<n>We train decoder-only transformers on vectors sampled from the exact distribution of random walks.<n>Across models and layers, the learned representations align strongly with the ground-truth predictive vectors and are often low-dimensional.<n>Although demonstrated in a simplified toy system, the analysis suggests that prefixes representations supporting optimal prediction may provide a useful lens for studying how neural networks internalize grammatical and other structural constraints.
  arXiv  Detail & Related papers  (2026-03-17T15:47:33Z)
• Depth-Wise Emergence of Prediction-Centric Geometry in Large Language Models [1.0742675209112622]
  We show that decoder-only large language models exhibit a transition from context-processing to prediction-forming phases of computation.<n>Using a unified framework combining geometric analysis with mechanistic intervention, we demonstrate that late-layer representations implement a structured geometric code that enables selective causal control over token prediction.
  arXiv  Detail & Related papers  (2026-02-04T11:00:16Z)
• Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation [85.82112629564942]
  We propose TokenBridge, which maintains the strong representation capacity of continuous tokens while preserving the modeling simplicity of discrete tokens.<n>We introduce a dimension-wise quantization strategy that independently discretizes each feature dimension, paired with a lightweight autoregressive prediction mechanism.<n>Our approach achieves reconstruction and generation quality on par with continuous methods while using standard categorical prediction.
  arXiv  Detail & Related papers  (2025-03-20T17:59:59Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [76.15163242945813]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence.<n>We introduce a novel generative model that generates tokens on the basis of human-interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• The Foundations of Tokenization: Statistical and Computational Concerns [51.370165245628975]
  Tokenization is a critical step in the NLP pipeline.<n>Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood.<n>The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models.
  arXiv  Detail & Related papers  (2024-07-16T11:12:28Z)
• TokenUnify: Scaling Up Autoregressive Pretraining for Neuron Segmentation [65.65530016765615]
  We propose a hierarchical predictive coding framework that captures multi-scale dependencies through three complementary learning objectives.<n> TokenUnify integrates random token prediction, next-token prediction, and next-all token prediction to create a comprehensive representational space.<n>We also introduce a large-scale EM dataset with 1.2 billion annotated voxels, offering ideal long-sequence visual data with spatial continuity.
  arXiv  Detail & Related papers  (2024-05-27T05:45:51Z)
• Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Prediction [79.78050867137594]
  Diffusion, masked-token prediction, and next-token prediction all use a Transformer network architecture. We analyze the scalability of each approach through the lens of compute budget measured in FLOPs. We find that token prediction methods, led by next-token prediction, significantly outperform diffusion on prompt following.
  arXiv  Detail & Related papers  (2024-05-21T21:49:39Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• Bayesian Graph Contrastive Learning [55.36652660268726]
  We propose a novel perspective of graph contrastive learning methods showing random augmentations leads to encoders. Our proposed method represents each node by a distribution in the latent space in contrast to existing techniques which embed each node to a deterministic vector. We show a considerable improvement in performance compared to existing state-of-the-art methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-12-15T01:45:32Z)
• Aligned Contrastive Predictive Coding [10.521845940927163]
  We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned.
  arXiv  Detail & Related papers  (2021-04-24T13:07:22Z)
• Latent Representation Prediction Networks [0.0]
  We find this principle of learning representations unsatisfying. We propose a new way of jointly learning this representation along with the prediction function. Our approach is shown to be more sample-efficient than standard reinforcement learning methods.
  arXiv  Detail & Related papers  (2020-09-20T14:26:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.