Predicting LLM Safety Before Release by Simulating Deployment

• URL: http://arxiv.org/abs/2607.07184v1
• Date: Wed, 08 Jul 2026 09:17:47 GMT
• Title: Predicting LLM Safety Before Release by Simulating Deployment
• Abstract summary: Pre-deployment safety evaluations aim to inform the downstream risks of releasing a new AI model.<n>Most evaluations provide limited evidence about how often undesired model behavior will occur in deployment.<n>We show that deployment simulation can be seeded from public chat datasets and remain informative about production misbehavior rates.
• Score: 5.628199352593384
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-deployment safety evaluations aim to inform the downstream risks of releasing a new AI model. Yet most evaluations provide limited evidence about how often undesired model behavior will occur in deployment: they generally have insufficient coverage, are unrepresentative, and are generally recognizable as tests. To address these concerns, we study a simple way to simulate a model deployment: starting from de-identified conversations from a previous model deployment, we hold fixed the initial conversation prefix and regenerate the next response using a candidate model. The resulting responses can then both be audited for novel misalignments and used to estimate the prevalence of model misbehavior before deployment. We evaluate deployment simulation across four GPT-5-series deployments, using registered, outcome-blinded predictions for GPT-5.4 and retrospective analyses of three earlier releases. We find that deployment simulation produces informative estimates of post-deployment misbehavior rates and outperforms baselines based on adversarially selected production data; its evaluation-awareness point estimates were also much closer to production traffic than those from traditional evaluations. We also identify the realism of tool resampling as a central challenge for further improving predictions and share results suggesting that this challenge is surmountable even in complex tool-use settings. Finally, we show that deployment simulation can be seeded from public chat datasets and remain informative about production misbehavior rates, suggesting a path for external researchers to run deployment-grounded evaluations without access to private production logs. Overall, deployment simulation helps evaluators forecast how language models will behave in the real world and supports more quantitative assessment of deployment risk.

Related papers

• Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds [35.96704508133401]
  Two techniques make simulated alignment evaluations harder to distinguish from real deployments.<n>Dish (Deployment-Imitating SWE-Agent Harness) wraps the target in an agent harness, reducing the gap between simulated and real deployment environments.<n>Results show that automated approaches can improve the realism of alignment evaluations.
  arXiv  Detail & Related papers  (2026-09-02T08:47:34Z)
• Data Leakage Inflates Generalizability of Power Outage Prediction Models [0.9558392439655014]
  We identify three common methodological choices in power outage prediction models that influence their ability to generalize.<n>We compare the predictive performance impacts of different methodological decisions using publicly available data for the U.S. East Coast from 2018 to 2023.<n>While random train-test splits yield strong performance, we show that these results are inflated by spatial and temporal autocorrelation.<n>These findings suggest that, given current data availability and evaluation practices, publicly trained outage prediction models offer limited and uncertain operational value.
  arXiv  Detail & Related papers  (2026-08-25T15:05:55Z)
• Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness [2.9267249513932008]
  We present a partial-identification approach to constructing bounds on the causal effect of a new model.<n>The core innovation in our approach is to leverage assumptions relating fine-grained predictive accuracy to downstream outcomes.
  arXiv  Detail & Related papers  (2026-07-23T20:45:10Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• From Data to Behavior: Predicting Unintended Model Behaviors Before Training [78.37660873165284]
  We introduce Data2Behavior, a new task for predicting unintended model behaviors prior to training.<n>We also propose Manipulating Data Features (MDF), a lightweight approach that summarizes candidate data through their mean representations.<n>Experiments on Qwen3-14B, Qwen2.5-32B-Instruct, and Gemma-3-12b-it confirm that MDF can anticipate unintended behaviors and provide insight into pre-training vulnerabilities.
  arXiv  Detail & Related papers  (2026-02-04T16:37:17Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• TRACE: A Self-Improving Framework for Robot Behavior Forecasting with Vision-Language Models [1.3408365072149797]
  Predicting the near-term behavior of a reactive agent is crucial in many robotic scenarios.<n>We present TRACE, an inference framework that couples tree-of-thought generation with domain-aware feedback.<n>We validate TRACE on both ground-vehicle simulations and real-world marine autonomous surface vehicles.
  arXiv  Detail & Related papers  (2025-03-02T06:58:02Z)
• Forecasting Rare Language Model Behaviors [20.712406244928832]
  We introduce a method to forecast potential risks across orders of magnitude more queries than we test during evaluation.<n>We find that our forecasts can predict the emergence of diverse undesirable behaviors across up to three orders of magnitude of query volume.<n>Our work enables model developers to proactively anticipate and patch rare failures before they manifest during large-scale deployments.
  arXiv  Detail & Related papers  (2025-02-24T03:16:15Z)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
  We introduce the first formal probabilistic evaluation framework for Large Language Models (LLMs)<n> Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model.<n>Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
  arXiv  Detail & Related papers  (2024-10-04T15:44:23Z)
• Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? [93.10694819127608]
  We propose a unified evaluation pipeline for forecasting methods with real-world perception inputs. Our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data.
  arXiv  Detail & Related papers  (2023-06-15T17:03:14Z)
• Scenario-based Evaluation of Prediction Models for Automated Vehicles [0.0]
  We argue that following evaluation practices in safety assessment for automated vehicles should be performed in a scenario-based fashion. We categorize trajectories of Open Motion dataset according to the type of movement they capture. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.
  arXiv  Detail & Related papers  (2022-10-11T09:45:43Z)
• A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks [72.7373468905418]
  We develop an open-source toolkit OpenBackdoor to foster the implementations and evaluations of textual backdoor learning. We also propose CUBE, a simple yet strong clustering-based defense baseline.
  arXiv  Detail & Related papers  (2022-06-17T02:29:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.