Quantifying non deterministic drift in large language models

• URL: http://arxiv.org/abs/2601.19934v1
• Date: Mon, 12 Jan 2026 10:34:48 GMT
• Title: Quantifying non deterministic drift in large language models
• Abstract summary: Large language models (LLMs) are widely used for tasks ranging from summarisation to decision support.<n>In practice, identical prompts do not always produce identical outputs, even when temperature and other decoding parameters are fixed.<n>We quantify behavioural baseline drift, defined as output variability observed when the same prompt is issued multiple times under operator-free conditions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are widely used for tasks ranging from summarisation to decision support. In practice, identical prompts do not always produce identical outputs, even when temperature and other decoding parameters are fixed. In this work, we conduct repeated-run experiments to empirically quantify baseline behavioural drift, defined as output variability observed when the same prompt is issued multiple times under operator-free conditions. We evaluate two publicly accessible models, gpt-4o-mini and llama3.1-8b, across five prompt categories using exact repeats, perturbed inputs, and reuse modes at temperatures of 0.0 and 0.7. Drift is measured using unique output fractions, lexical similarity, and word count statistics, enabling direct comparison across models, prompting modes, and deployment types. The results show that nondeterminism persists even at temperature 0.0, with distinct variability patterns by model size, deployment, and prompt type. We situate these findings within existing work on concept drift, behavioural drift, and infrastructure-induced nondeterminism, discuss the limitations of lexical metrics, and highlight emerging semantic approaches. By establishing a systematic empirical baseline in the absence of stabilisation techniques, this study provides a reference point for evaluating future drift mitigation and control methods.

Related papers

• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• Robust Confinement State Classification with Uncertainty Quantification through Ensembled Data-Driven Methods [39.27649013012046]
  We develop methods for confinement state classification with uncertainty quantification and model robustness.<n>We focus on off-line analysis for TCV discharges, distinguishing L-mode, H-mode, and an in-between dithering phase (D)<n>A dataset of 302 TCV discharges is fully labeled, and will be publicly released.
  arXiv  Detail & Related papers  (2025-02-24T18:25:22Z)
• Distinguishing Cause from Effect with Causal Velocity Models [3.0523869645673076]
  We develop a method for causal discovery that extends beyond known model classes such as additive or location scale noise.<n>When the score is estimated well, the objective is also useful for detecting model non-identifiability and misspecification.
  arXiv  Detail & Related papers  (2025-02-07T17:50:14Z)
• On the Identification of Temporally Causal Representation with Instantaneous Dependence [50.14432597910128]
  Temporally causal representation learning aims to identify the latent causal process from time series observations. Most methods require the assumption that the latent causal processes do not have instantaneous relations. We propose an textbfIDentification framework for instantanetextbfOus textbfLatent dynamics.
  arXiv  Detail & Related papers  (2024-05-24T08:08:05Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• On the Impact of Temporal Concept Drift on Model Explanations [31.390397997989712]
  Explanation faithfulness of model predictions in natural language processing is evaluated on held-out data from the same temporal distribution as the training data. We examine the impact of temporal variation on model explanations extracted by eight feature attribution methods and three select-then-predict models across six text classification tasks.
  arXiv  Detail & Related papers  (2022-10-17T15:53:09Z)
• Continuous-Time Modeling of Counterfactual Outcomes Using Neural Controlled Differential Equations [84.42837346400151]
  Estimating counterfactual outcomes over time has the potential to unlock personalized healthcare. Existing causal inference approaches consider regular, discrete-time intervals between observations and treatment decisions. We propose a controllable simulation environment based on a model of tumor growth for a range of scenarios.
  arXiv  Detail & Related papers  (2022-06-16T17:15:15Z)
• Suitability of Different Metric Choices for Concept Drift Detection [9.76294323004155]
  Many unsupervised approaches for drift detection rely on measuring the discrepancy between the sample of two time windows. Most drift detection methods can be distinguished in what metric they use, how this metric is estimated, and how the decision threshold is found. We compare different types of estimators and metrics theoretically and empirically and investigate the relevance of the single metric components.
  arXiv  Detail & Related papers  (2022-02-19T01:11:32Z)
• Leveraging Global Parameters for Flow-based Neural Posterior Estimation [90.21090932619695]
  Inferring the parameters of a model based on experimental observations is central to the scientific method. A particularly challenging setting is when the model is strongly indeterminate, i.e., when distinct sets of parameters yield identical observations. We present a method for cracking such indeterminacy by exploiting additional information conveyed by an auxiliary set of observations sharing global parameters.
  arXiv  Detail & Related papers  (2021-02-12T12:23:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.