Adoption, usability and perceived clinical value of a UK AI clinical reference platform (iatroX): a mixed-methods formative evaluation of real-world usage and a 1,223-respondent user survey

• URL: http://arxiv.org/abs/2509.21188v1
• Date: Thu, 25 Sep 2025 14:03:04 GMT
• Title: Adoption, usability and perceived clinical value of a UK AI clinical reference platform (iatroX): a mixed-methods formative evaluation of real-world usage and a 1,223-respondent user survey
• Abstract summary: Clinicians face growing information overload from biomedical literature and guidelines, hindering evidence-based care.<n>We describe iatroX, a UK-centred RAG-based clinical reference platform.<n>We report early adoption, usability, and perceived clinical value from a formative implementation evaluation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinicians face growing information overload from biomedical literature and guidelines, hindering evidence-based care. Retrieval-augmented generation (RAG) with large language models may provide fast, provenance-linked answers, but requires real-world evaluation. We describe iatroX, a UK-centred RAG-based clinical reference platform, and report early adoption, usability, and perceived clinical value from a formative implementation evaluation. Methods comprised a retrospective analysis of usage across web, iOS, and Android over 16 weeks (8 April-31 July 2025) and an in-product intercept survey. Usage metrics were drawn from web and app analytics with bot filtering. A client-side script randomized single-item prompts to approx. 10% of web sessions from a predefined battery assessing usefulness, reliability, and adoption intent. Proportions were summarized with Wilson 95% confidence intervals; free-text comments underwent thematic content analysis. iatroX reached 19,269 unique web users, 202,660 engagement events, and approx. 40,000 clinical queries. Mobile uptake included 1,960 iOS downloads and Android growth (peak >750 daily active users). The survey yielded 1,223 item-level responses: perceived usefulness 86.2% (95% CI 74.8-93.9%; 50/58); would use again 93.3% (95% CI 68.1-99.8%; 14/15); recommend to a colleague 88.4% (95% CI 75.1-95.9%; 38/43); perceived accuracy 75.0% (95% CI 58.8-87.3%; 30/40); reliability 79.4% (95% CI 62.1-91.3%; 27/34). Themes highlighted speed, guideline-linked answers, and UK specificity. Early real-world use suggests iatroX can mitigate information overload and support timely answers for UK clinicians. Limitations include small per-item samples and early-adopter bias; future work will include accuracy audits and prospective studies on workflow and care quality.

Related papers

• Enhancing Clinician Decision-Making via Uncertainty-Aware Multi-Expert Fusion for Stroke Rehabilitation [0.32553561239735207]
  We present xAARA: an engine designed to augment rather than replace clinical judgment.<n>xAARA returns ARAT assessments with calibrated uncertainty and explanations across task, movement-phase, and movement-quality levels.<n>In 105 stroke survivors (788 exercises), xAARA achieved 94.2% task accuracy and 81.3% movement-phase accuracy.
  arXiv  Detail & Related papers  (2026-06-23T09:00:15Z)
• Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agents [51.46570338144688]
  ProAct is a proactive agent architecture that leverages idle-time compute to anticipate and fulfill likely upcoming user needs.<n>By analyzing evolving dialogue history together with persistent memory, ProAct predicts upcoming needs and iteratively acquires information.<n>MemBench evaluations confirm that ProAct achieves state-of-the-art reflective accuracy, underscoring its sustained and robust performance.
  arXiv  Detail & Related papers  (2026-05-25T15:47:21Z)
• Automating Clinical Information Retrieval from Finnish Electronic Health Records Using Large Language Models [0.0]
  Locally deployable framework answers clinical questions directly from EHRs without external data transfer.<n>Open-source large language models (LLMs) ranging from 4B to 70B parameters were benchmarked under fully offline conditions.
  arXiv  Detail & Related papers  (2026-03-27T14:03:51Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems [19.880569341968023]
  Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety.<n>We propose a retrieval-augmented multi-agent framework designed to automate the generation of instance-specific evaluation rubrics.
  arXiv  Detail & Related papers  (2026-01-21T16:40:41Z)
• MLB: A Scenario-Driven Benchmark for Evaluating Large Language Models in Clinical Applications [27.73095565539546]
  We introduce a Medical LLM Benchmark MLB, a benchmark evaluating Large Language Models (LLMs) on both foundational knowledge and scenario-based reasoning.<n>MLB is structured around five core dimensions: Medical Knowledge (MedKQA), Safety and Ethics (MedSE), Medical Record Understanding (MedRU), Smart Services (SmartServ), and Smart Healthcare (SmartCare)<n>Its design features a rigorous curation pipeline involving 300 licensed physicians. Besides, we provide a scalable evaluation methodology, centered on a specialized judge model trained via Supervised Fine-Tuning (SFT) on expert annotations.
  arXiv  Detail & Related papers  (2026-01-08T02:41:42Z)
• Explainable Admission-Level Predictive Modeling for Prolonged Hospital Stay in Elderly Populations: Challenges in Low- and Middle-Income Countries [65.4286079244589]
  Prolonged length of stay (pLoS) is a significant factor associated with the risk of adverse in-hospital events.<n>We develop and explain a predictive model for pLos using admission-level patient and hospital administrative data.
  arXiv  Detail & Related papers  (2026-01-07T23:35:24Z)
• Beyond Long Context: When Semantics Matter More than Tokens [0.0]
  Electronic Health Records store clinical documentation as base64 attachments encoded in FHIR DocumentReference resources.<n>The Clinical Entity Augmented Retrieval (CLEAR) method, introduced by Lopez et al. 2025, uses entity aware retrieval.<n>CLEAR achieved a 58.3 percent win rate, an average semantic similarity of 0.878, and used 78 percent fewer tokens than wide context processing.
  arXiv  Detail & Related papers  (2025-10-29T16:41:44Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines [1.9615061725959186]
  This paper presents the development and evaluation of a Retrieval-Augmented Generation system for querying the United Kingdom's National Institute for Health and Care Excellence (NICE) clinical guidelines using Large Language Models (LLMs)<n>The system's retrieval architecture, composed of a hybrid embedding mechanism, was evaluated against a database of 10,195 text chunks derived from three hundred guidelines.<n>It demonstrates high performance, with a Mean Reciprocal Rank (MRR) of 0.814, a Recall of 81% at the first chunk and of 99.1% within the top ten retrieved chunks, when evaluated on 7901 queries.
  arXiv  Detail & Related papers  (2025-10-03T12:57:13Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• Evaluating Large Language Models for Evidence-Based Clinical Question Answering [4.101088122511548]
  Large Language Models (LLMs) have demonstrated substantial progress in biomedical and clinical applications.<n>We curate a benchmark drawing from Cochrane systematic reviews and clinical guidelines.<n>We observe consistent performance patterns across sources and clinical domains.
  arXiv  Detail & Related papers  (2025-09-13T15:03:34Z)
• An Agentic System for Rare Disease Diagnosis with Traceable Reasoning [69.46279475491164]
  We introduce DeepRare, the first rare disease diagnosis agentic system powered by a large language model (LLM)<n>DeepRare generates ranked diagnostic hypotheses for rare diseases, each accompanied by a transparent chain of reasoning.<n>The system demonstrates exceptional diagnostic performance among 2,919 diseases, achieving 100% accuracy for 1013 diseases.
  arXiv  Detail & Related papers  (2025-06-25T13:42:26Z)
• A Clinician-Friendly Platform for Ophthalmic Image Analysis Without Technical Barriers [51.45596445363302]
  GlobeReady is a clinician-friendly AI platform that enables fundus disease diagnosis without retraining, fine-tuning, or the needs for technical expertise.<n>We demonstrate high accuracy across imaging modalities: 93.9-98.5% for 11 fundus diseases using color fundus photographs (CPFs) and 87.2-92.7% for 15 fundus diseases using optic coherence tomography ( OCT) scans.<n>By leveraging training-free local feature augmentation, GlobeReady platform effectively mitigates domain shifts across centers and populations.
  arXiv  Detail & Related papers  (2025-04-22T14:17:22Z)
• Accessible, At-Home Detection of Parkinson's Disease via Multi-task Video Analysis [3.1851272788128644]
  Existing AI-based Parkinson's Disease detection methods primarily focus on unimodal analysis of motor or speech tasks.<n>We propose a novel Uncertainty-calibrated Fusion Network (UFNet) that leverages this multimodal data to enhance diagnostic accuracy.<n>UFNet significantly outperformed single-task models in terms of accuracy, area under the ROC curve (AUROC), and sensitivity while maintaining non-inferior specificity.
  arXiv  Detail & Related papers  (2024-06-21T04:02:19Z)
• Early prediction of onset of sepsis in Clinical Setting [0.8471078314535754]
  A supervised learning approach was adopted, wherein an XGBoost model was trained utilizing 80% of the train dataset. The model was validated on prospective data that was entirely unseen during the training phase. The model achieved a normalized utility score of 0.494 on test data and 0.378 on prospective data at threshold 0.3.
  arXiv  Detail & Related papers  (2024-02-05T19:58:40Z)
• Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review [0.0]
  This study aims to enhance the clinical use of automated sleep-scoring algorithms by incorporating an uncertainty estimation approach. Total of 19578 PSGs from 13 open-access databases were used to train U-Sleep, a state-of-the-art sleep-scoring algorithm.
  arXiv  Detail & Related papers  (2023-12-22T15:58:09Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Sentiment Analysis of Users' Reviews on COVID-19 Contact Tracing Apps with a Benchmark Dataset [6.592595861973966]
  Contact tracing has been globally adopted in the fight to control the infection rate of COVID-19. Thanks to digital technologies, such as smartphones and wearable devices, contacts of COVID-19 patients can be easily traced and informed about their potential exposure to the virus. Several interesting mobile applications have been developed. However, there are ever-growing concerns over the working mechanism and performance of these applications. In this work, we propose a pipeline starting from manual annotation via a crowd-sourcing study and concluding on the development and training of AI models for automatic sentiment analysis of users' reviews.
  arXiv  Detail & Related papers  (2021-03-01T18:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.