From Simulation to Enaction: Post-trained language models recognize and react to their own generations

• URL: http://arxiv.org/abs/2605.25459v1
• Date: Mon, 25 May 2026 06:09:04 GMT
• Title: From Simulation to Enaction: Post-trained language models recognize and react to their own generations
• Abstract summary: Language models are pretrained as passive predictors with no incentive to model the consequences of their own outputs.<n>We present evidence that post-trained models recognize their on-policy generations, and this recognition is implicitly encoded in their output distributions.
• Score: 1.9010546392162666
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models are pretrained as passive predictors with no incentive to model the consequences of their own outputs. Post-training changes this: a model producing its own responses can benefit from recognizing that it is on-policy. We present evidence that post-trained models recognize their on-policy generations, and this recognition is implicitly encoded in their output distributions. In particular, on-policy output distribution entropy is 3--4$\times$ lower than off-policy entropy, across model families and size classes. We trace part of this effect to an internal representation of input surprise, tracking the unlikeliness of the most recent input token according to the model's prior predictions, that causally modulates output entropy. One example of these phenomena can be observed in response to open-ended prompts; post-trained models (unlike pretrained models) collapse their uncertainty over the topic of their upcoming response before the first output token; violating this cached intention with a different-topic prefill results in higher output entropy. We also tested whether models can distinguish on-policy contexts from prefills via explicit verbal report. We find that they can, but that interestingly, this explicit recognition routes through a different mechanism than implicit recognition.

Related papers

• Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controls [30.071130311851277]
  A model's behavior on a task is jointly determined by the input it receives and the prior it brings in, i.e. the distribution over stimuli it implicitly expects.<n>We propose a method to sample from models' perceptual prior distributions directly, by steering a generative model to produce stimuli along controllable axes and running Gibbs sampling over that space with the model under study as the judge.
  arXiv  Detail & Related papers  (2026-08-27T21:38:41Z)
• Counterfactual Quotient Models: Learning What Actions Change, Not What the World Does [14.67340173071927]
  Reinforcement-learning models commonly predict complete future states, observations, or feature occupancies.<n>We introduce the Counterfactual Quotient Model, which treats action-conditioned futures as equivalent when they differ only by a component shared across actions.
  arXiv  Detail & Related papers  (2026-08-22T19:55:57Z)
• Know Your Limits: Entropy Estimation Modeling for Compression and Generalization [0.0]
  We introduce encoder-augmented causal decoder model architectures that exhibit superior training efficiency characteristics.<n>We show that causal models trained to approach but not exceed estimated per-token entropies exhibit greater generalization than models trained without taking entropy into account.
  arXiv  Detail & Related papers  (2025-11-13T18:46:42Z)
• Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors [61.92704516732144]
  We show that the most robust features for correctness prediction are those that play a distinctive causal role in the model's behavior.<n>We propose two methods that leverage causal mechanisms to predict the correctness of model outputs.
  arXiv  Detail & Related papers  (2025-05-17T00:31:39Z)
• Language Models Can Predict Their Own Behavior [29.566208688211876]
  Language models (LMs) can exhibit specific behaviors,' such as a failure to follow alignment training, that we hope to detect and react to during deployment.<n>We show that probes trained on the internal representation of input tokens alone can predict a wide range of eventual behaviors over the entire output sequence.<n>An early warning system built on the probes reduces jailbreaking by 91%.
  arXiv  Detail & Related papers  (2025-02-18T23:13:16Z)
• Heat Death of Generative Models in Closed-Loop Learning [63.83608300361159]
  We study the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. We show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to degenerate.
  arXiv  Detail & Related papers  (2024-04-02T21:51:39Z)
• Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training [57.771940716189114]
  We show that large language models (LLMs) suffer from the "reversal curse" The root cause of the reversal curse lies in the different word order between the training and inference stage. We propose Semantic-aware Permutation Training (SPT) to address this issue.
  arXiv  Detail & Related papers  (2024-03-01T18:55:20Z)
• Limitations of Agents Simulated by Predictive Models [1.6649383443094403]
  We outline two structural reasons for why predictive models can fail when turned into agents. We show that both of those failures are fixed by including a feedback loop from the environment. Our treatment provides a unifying view of those failure modes, and informs the question of why fine-tuning offline learned policies with online learning makes them more effective.
  arXiv  Detail & Related papers  (2024-02-08T17:08:08Z)
• Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions [9.909170013118775]
  This work presents a linear decomposition of final hidden states from autoregressive language models based on each initial input token. Using the change in next-word probability as a measure of importance, this work first examines which context words make the biggest contribution to language model predictions.
  arXiv  Detail & Related papers  (2023-05-17T23:55:32Z)
• NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries? [61.58261351116679]
  We introduce a two-stage adversarial example generation framework (NaturalAdversaries) for natural language understanding tasks. It is adaptable to both black-box and white-box adversarial attacks based on the level of access to the model parameters. Our results indicate these adversaries generalize across domains, and offer insights for future research on improving robustness of neural text classification models.
  arXiv  Detail & Related papers  (2022-11-08T16:37:34Z)
• Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
  We show that pre-trained language models with few examples show strong prediction bias across labels. Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features. These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
  arXiv  Detail & Related papers  (2022-04-17T15:55:18Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• BERT & Family Eat Word Salad: Experiments with Text Understanding [17.998891912502092]
  We study the response of large models from the BERT family to incoherent inputs that should confuse any model that claims to understand natural language. Experiments show that state-of-the-art models consistently fail to recognize them as ill-formed, and instead produce high confidence predictions on them. We show that if models are explicitly trained to recognize invalid inputs, they can be robust to such attacks without a drop in performance.
  arXiv  Detail & Related papers  (2021-01-10T01:32:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.