The Token Not Taken: Sampling, State, and the Variability of AI Agent Outputs

• URL: http://arxiv.org/abs/2606.08998v1
• Date: Mon, 08 Jun 2026 03:53:55 GMT
• Title: The Token Not Taken: Sampling, State, and the Variability of AI Agent Outputs
• Abstract summary: Agentic AI systems can behave differently across runs.<n> variability arises from several layers that are often conflated.<n>This manuscript clarifies what it means to call agentic AI systems, when such variability can be reproduced under matched conditions.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agentic AI systems can behave differently across runs: the same request may produce a different plan, a different tool call, a different code edit, or a different final answer. Such variability arises from several layers that are often conflated. A foundation model is a large pretrained model, usually adaptable to many downstream tasks, that maps an input context to predictions over outputs. In many current agents, that model is embedded in an orchestration loop that plans, calls tools, observes results, and updates state. One explicit intrinsic source of variability in such systems is token generation: the model computes scores over possible next tokens, the scores are converted into probabilities, and a decoder may sample tokens using a pseudo-random number generator. A small sampled token difference can then propagate upward into a different tool call, code path, search query, or agent state. Other sources of variability are extrinsic to token sampling, including changing environments, live data, serving infrastructure, batch effects, and numerical details. By separating these layers, the manuscript clarifies what it means to call agentic AI systems stochastic, when such variability can be reproduced under matched conditions, and why deterministic execution need not imply identical behavior in deployed settings.

Related papers

• GenOS: Compositional Certificates for Semantic Robustness in AI Code Generation [0.0]
  GenOS is a probabilistic operational semantics for replacing a prompt, contract, generator, or program.<n>We prove that equivalence-compatible kernels descend to quotient classes and that quotienting commutes with distributional extension and sequential composition.<n>We also establish workflow bi-commit safety under sound validation, total-variation non-expansiveness, and an additive bound.
  arXiv  Detail & Related papers  (2026-08-04T12:42:11Z)
• Graphical conditional generative modeling for digital twin modeling [7.0264064660746754]
  We introduce a framework for building parsimonious surrogate models from observational data.<n>We identify which candidate inputs influence the full conditional law of a target quantity, rather than only its conditional mean.<n>In control settings, the method identifies not only a transition model, but also the state action, and memory variables needed to make the learned dynamics effectively Markovian.
  arXiv  Detail & Related papers  (2026-06-15T04:56:38Z)
• Conditional Attribute Estimation with Autoregressive Sequence Models [2.956641142618867]
  Conditional Attribute Transformers is a novel method for jointly estimating the next-token probability and the value of an attribute on each potential next token selection.<n>Our approach achieves state of the art performance on sparse reward tasks, estimates attribute probabilities orders of magnitude faster than sampling, and can guide decoding of autoregressive sequence models on a range of language tasks.
  arXiv  Detail & Related papers  (2026-05-13T18:11:16Z)
• AutoStan: Autonomous Bayesian Model Improvement via Predictive Feedback [0.38073142980733005]
  AutoStan is a framework in which a command-line interface (CLI) coding agent autonomously builds and iteratively improves Bayesian models written in Stan.<n>We evaluate AutoStan on five datasets with diverse modeling structures.
  arXiv  Detail & Related papers  (2026-03-29T16:58:46Z)
• Deep Multivariate Models with Parametric Conditionals [47.20275199636936]
  We consider deep multivariate models for heterogeneous collections of random variables.<n>We propose to represent the joint probability distribution by means of conditional probability distributions for each group of variables conditioned on the rest.<n>Their learning can be approached as training a parametrised Markov chain kernel by maximising the data likelihood of its limiting distribution.
  arXiv  Detail & Related papers  (2026-02-02T11:01:48Z)
• Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention [0.0]
  Zonkey is a hierarchical diffusion model that addresses limitations through a fully trainable pipeline from raw characters to document-level representations.<n>At its core is a differentiable tokenizer that learns probabilistic beginning-of-sequence (BOS) decisions.<n>Zonkey generates coherent, variable-length text from noise, demonstrating emergent hierarchies.
  arXiv  Detail & Related papers  (2026-01-29T14:17:37Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Multi-Candidate Speculative Decoding [82.05519287513444]
  Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments from a fast draft model that is then verified in parallel by the target model. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model.
  arXiv  Detail & Related papers  (2024-01-12T17:15:23Z)
• CREATED: Generating Viable Counterfactual Sequences for Predictive Process Analytics [0.30586855806896046]
  We propose a general framework that uses evolutionary methods to generate counterfactual sequences. Our framework does not require domain knowledge. Instead, we propose to train a Markov model to compute the feasibility of generated counterfactual sequences.
  arXiv  Detail & Related papers  (2023-03-28T09:35:25Z)
• IOT: Instance-wise Layer Reordering for Transformer Structures [173.39918590438245]
  We break the assumption of the fixed layer order in the Transformer and introduce instance-wise layer reordering into the model structure. Our method can also be applied to other architectures beyond Transformer.
  arXiv  Detail & Related papers  (2021-03-05T03:44:42Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• Unsupervised Controllable Generation with Self-Training [90.04287577605723]
  controllable generation with GANs remains a challenging research problem. We propose an unsupervised framework to learn a distribution of latent codes that control the generator through self-training. Our framework exhibits better disentanglement compared to other variants such as the variational autoencoder.
  arXiv  Detail & Related papers  (2020-07-17T21:50:35Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.