Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

• URL: http://arxiv.org/abs/2512.15605v1
• Date: Wed, 17 Dec 2025 17:14:26 GMT
• Title: Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction
• Abstract summary: We provide a unified view of autoregressive models (ARMs) and energy-based models (EBMs)<n>We derive the equivalence between supervised learning of ARMs and EBMs.<n>Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.
• Score: 13.764740803730803
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Autoregressive models (ARMs) currently constitute the dominant paradigm for large language models (LLMs). Energy-based models (EBMs) represent another class of models, which have historically been less prevalent in LLM development, yet naturally characterize the optimal policy in post-training alignment. In this paper, we provide a unified view of these two model classes. Taking the chain rule of probability as a starting point, we establish an explicit bijection between ARMs and EBMs in function space, which we show to correspond to a special case of the soft Bellman equation in maximum entropy reinforcement learning. Building upon this bijection, we derive the equivalence between supervised learning of ARMs and EBMs. Furthermore, we analyze the distillation of EBMs into ARMs by providing theoretical error bounds. Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.

Related papers

• The Efficiency Gap in Byte Modeling [101.18202046105718]
  Two alternative paradigms have challenged the use of subword tokenization and autoregressive ordering.<n>We show that removing these structural priors incurs a significant computational cost.<n>Our findings suggest that future modality-agnostic designs must incorporate alternative structural biases to maintain viable scaling trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:03:30Z)
• Simulated Annealing Enhances Theory-of-Mind Reasoning in Autoregressive Language Models [1.4323566945483497]
  Theory of Mind (ToM) tasks crucially depend on reasoning about latent mental states of oneself and others.<n>We show that strong ToM capability can be recovered directly from the base model without any additional weight updates or verifications.
  arXiv  Detail & Related papers  (2026-01-18T05:51:30Z)
• Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries [3.930598942647121]
  We propose a two-stage LM-based judging reward model that utilizes an explanation based slot framework for prediction.<n>In both reinforcement learning from human feedback (RLHF) and out-of-distribution (OOD) scenarios, the ESFP-RM framework delivers more stable and generalizable reward signals.
  arXiv  Detail & Related papers  (2025-08-25T17:11:28Z)
• Towards Machine Theory of Mind with Large Language Model-Augmented Inverse Planning [0.022940141855172035]
  We propose a hybrid approach to machine Theory of Mind (ToM) that uses large language models (LLMs) as a mechanism for generating hypotheses and likelihood functions.<n>We also exhibit the model's potential to predict mental states on open-ended tasks.
  arXiv  Detail & Related papers  (2025-07-04T16:01:27Z)
• Large Language Diffusion Models [93.26422905620008]
  Large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs)<n>We introduce LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning paradigm.<n>Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines.
  arXiv  Detail & Related papers  (2025-02-14T08:23:51Z)
• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step.<n>Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space [72.52365911990935]
  We introduce Bellman Diffusion, a novel DGM framework that maintains linearity in MDPs through gradient and scalar field modeling. Our results show that Bellman Diffusion achieves accurate field estimations and is a capable image generator, converging 1.5x faster than the traditional histogram-based baseline in distributional RL tasks.
  arXiv  Detail & Related papers  (2024-10-02T17:53:23Z)
• Predictable MDP Abstraction for Unsupervised Model-Based RL [93.91375268580806]
  We propose predictable MDP abstraction (PMA) Instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches.
  arXiv  Detail & Related papers  (2023-02-08T07:37:51Z)
• Your Autoregressive Generative Model Can be Better If You Treat It as an Energy-Based One [83.5162421521224]
  We propose a unique method termed E-ARM for training autoregressive generative models. E-ARM takes advantage of a well-designed energy-based learning objective. We show that E-ARM can be trained efficiently and is capable of alleviating the exposure bias problem.
  arXiv  Detail & Related papers  (2022-06-26T10:58:41Z)
• Generative Temporal Difference Learning for Infinite-Horizon Prediction [101.59882753763888]
  We introduce the $gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. We discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors.
  arXiv  Detail & Related papers  (2020-10-27T17:54:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.