Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System

• URL: http://arxiv.org/abs/2606.12702v1
• Date: Wed, 10 Jun 2026 21:44:20 GMT
• Title: Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System
• Abstract summary: Large language models (LLMs) are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems.<n> static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets.<n>We perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions.
• Score: 33.34100218601891
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets -- leading to major blind spots for evaluating clinical systems. In this work, we perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions. Specifically, we train a pre-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment-specific context available before generation. We conduct a prospective analysis of our model over 4.5 months of user feedback, finding that our prediction model achieves an AUROC of 0.719. Further, we estimate the benefit of such predictions in two downstream use cases (guardrail triggering and abstention). Our key conceptual insight is that making use of deployment-specific context (i.e., the provider type, department name, language model used for response), as opposed to only query content, improves the ability to predict whether the user will reject the system output. Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment-specific context, opening the door to targeted guardrails.

Related papers

• LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications [76.41731220830714]
  Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tools, and iterative prediction.<n>We examine applications in finance, weather, health, energy, and operations, and we summarize the benchmarks and datasets used for evaluation.
  arXiv  Detail & Related papers  (2026-08-24T10:02:39Z)
• Rethinking the Evaluation and Optimization of LLM-Based Social Simulation [56.18577997572273]
  We use the subjectivity coefficient to analyze how accuracy-based evaluation and hard-label training fail as subjectivity grows.<n>We construct SUBJSIM, a benchmark of 19,300 contexts covering 193 annotators and 100 subjective questions.<n>Results on SUBJSIM demonstrate the advantages of our method.
  arXiv  Detail & Related papers  (2026-08-20T06:26:03Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• PREBA: Surgical Duration Prediction via PCA-Weighted Retrieval-Augmented LLMs and Bayesian Averaging Aggregation [51.96735866702332]
  PREBA is a retrieval-augmented framework that integrates PCA-weighted retrieval and Bayesian averaging aggregation.<n> PREBA significantly improves performance-for instance, reducing MAE by up to 40% and raising R2 from -0.13 to 0.62 over zero-shot inference.
  arXiv  Detail & Related papers  (2026-02-27T07:19:23Z)
• Context-Adaptive Requirements Defect Prediction through Human-LLM Collaboration [1.4499356176178066]
  We propose a Human-LLM Collaboration (HLC) approach that treats defect prediction as an adaptive process rather than a static classification task.<n>We evaluate this approach using the weak word smell on the QuRE benchmark of 1,266 annotated Mercedes-Benz requirements.
  arXiv  Detail & Related papers  (2026-01-05T10:00:14Z)
• Bayesian Evaluation of Large Language Model Behavior [11.847752638476257]
  It is increasingly important to evaluate how text generation systems based on large language models behave.<n>Existing approaches to evaluation often neglect statistical uncertainty quantification.<n>We present two case studies applying a Bayesian approach for quantifying uncertainty in binary evaluation metrics.
  arXiv  Detail & Related papers  (2025-11-04T19:51:46Z)
• Clinical Uncertainty Impacts Machine Learning Evaluations [40.773483049446426]
  We argue that machine-learning evaluations should explicitly account for annotation uncertainty using probabilistic metrics that directly operate on distributions.<n>We urge the community to release raw annotations for datasets and to adopt uncertainty-aware evaluation so that performance estimates may better reflect clinical data.
  arXiv  Detail & Related papers  (2025-09-26T11:56:58Z)
• Aligning the Evaluation of Probabilistic Predictions with Downstream Value [2.6636053598505307]
  Metrics based solely on predictive performance often diverge from measures of real-world downstream impact.<n>We propose a data-driven method to learn a proxy evaluation function aligned with the downstream evaluation.<n>Our approach leverages weighted scoring rules parametrized by a neural network, where weighting is learned to align with the performance in the downstream task.
  arXiv  Detail & Related papers  (2025-08-25T17:41:27Z)
• Stream-Based Monitoring of Algorithmic Fairness [4.811789437743092]
  Stream-based monitoring is proposed as a solution for verifying the algorithmic fairness of decision and prediction systems at runtime.<n>We present a principled way to formalize algorithmic fairness over temporal data streams in the specification language RTLola.
  arXiv  Detail & Related papers  (2025-01-30T13:18:59Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.