Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions

• URL: http://arxiv.org/abs/2608.13786v1
• Date: Thu, 13 Aug 2026 21:38:01 GMT
• Title: Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions
• Abstract summary: ChatGPT, Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5 were evaluated.<n>We prompted the models with clinical questions adapted from 20 review questions in Issues 6 and 7 of the 2026 Cochrane Database of Systematic Reviews.<n>A response retrieved 39.2% $pm$ 29.8% of Cochrane included studies, while citing 5.0% $pm$ 9.4% of excluded studies.
• Score: 9.17137908309045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) chatbots are increasingly used to answer clinical questions with citations to relevant clinical studies. Prior research has largely focused on citation fabrication, leaving a gap in evaluating the quality of retrieved studies and the factors driving their selection. In this study, we evaluated three general-purpose LLM chatbots: Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5. We prompted the models with clinical questions adapted from 20 review questions in Issues 6 and 7 of the 2026 Cochrane Database of Systematic Reviews, simulating patient, clinician, and evidence-synthesis researcher roles. Each chatbot was queried under each user role with four independent repetitions, yielding 720 responses. Each chatbot was asked to support its answers with primary clinical citations, which we benchmarked against the included and excluded study sets of the Cochrane reviews. On average, a chatbot response retrieved 39.2% $\pm$ 29.8% of Cochrane included studies, while citing 5.0% $\pm$ 9.4% of excluded studies. Recall of Cochrane included studies varied significantly by model and user role. ChatGPT achieved higher recall than Claude or Gemini (63.1% $\pm$ 29.5% vs. 37.0% $\pm$ 23.8% vs. 17.3% $\pm$ 13.1%; $p=2.0\times10^{-5}$). The researcher role yielded higher recall than the clinician or patient roles (42.8% $\pm$ 30.8% vs. 38.6% $\pm$ 28.9% vs. 36.1% $\pm$ 29.3%; $p=2.0\times10^{-5}$). Controlling for publication year, citations per year, and open-access status, sample size was the only independently significant predictor of retrieval (odds ratio 1.80 per 1-unit increase in log sample size, 95% CI 1.37-2.36, $p=2.34\times10^{-5}$). These findings suggest that while LLM chatbots can retrieve some studies identified by expert reviewers, their performance varies by model and user role, and they exhibit a bias toward clinical trials with larger sample sizes.

Related papers

• Evaluating Commercial AI Chatbots as News Intermediaries [85.32040752972836]
  Best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier.<n>Every model achieves its lowest accuracy on Hindi.<n> retrieval, not reasoning, failures drive over 70% of all errors.
  arXiv  Detail & Related papers  (2026-05-21T17:42:07Z)
• ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV [0.0]
  Reasoning benchmarks measure clinical performance on clean inputs.<n>We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one.<n>EpiKG carries assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent.
  arXiv  Detail & Related papers  (2026-05-11T18:47:52Z)
• Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness [49.2667937337333]
  We stress-test this assumption for detecting incomplete patient-facing medical responses.<n>We evaluate three granularities (General-Likert, Analytical-Rubric, Dynamic-Checklist) and three backbone models across two clinician-annotated datasets.
  arXiv  Detail & Related papers  (2026-03-26T19:01:55Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• AI chatbots versus human healthcare professionals: a systematic review and meta-analysis of empathy in patient care [0.10262304700896197]
  Use of artificial intelligence (AI)-based chatbots in healthcare is rapidly expanding.<n>Some studies suggest AI chatbots can outperform human healthcare professionals (HCPs) in empathy.<n>We searched multiple databases for studies comparing AI chatbots with human HCPs on empathy measures.<n>Thirteen studies reported statistically significantly higher empathy ratings for AI, with only two studies situated in dermatology favouring human responses.
  arXiv  Detail & Related papers  (2026-02-05T13:09:19Z)
• Large language models provide unsafe answers to patient-posed medical questions [0.12568469427065204]
  We compare the safety of four publicly available chatbots--Claude by Anthropic, Gemini by Google, GPT-4o by OpenAI, and Llama3-70B by Meta--on a new dataset, HealthAdvice.<n>The rate of problematic responses varies from 21.6 percent (Claude) to 43.2 percent (Llama), with unsafe responses varying from 5 percent (Claude) to 13 percent (GPT-4o, Llama)
  arXiv  Detail & Related papers  (2025-07-25T02:55:36Z)
• Development and Evaluation of HopeBot: an LLM-based chatbot for structured and interactive PHQ-9 depression screening [48.355615275247786]
  HopeBot administers the Patient Health Questionnaire-9 (PHQ-9) using retrieval-augmented generation and real-time clarification.<n>In a within-subject study, 132 adults in the United Kingdom and China completed both self-administered and chatbots versions.<n>Overall, 87.1% expressed willingness to reuse or recommend HopeBot.
  arXiv  Detail & Related papers  (2025-07-08T13:41:22Z)
• Quriosity: Analyzing Human Questioning Behavior and Causal Inquiry through Curiosity-Driven Queries [92.1651731484397]
  We present Quriosity, a collection of 13.5K naturally occurring questions from three diverse sources.<n>Our analysis reveals a significant presence of causal questions (up to 42%) in the dataset.
  arXiv  Detail & Related papers  (2024-05-30T17:55:28Z)
• Performance of ChatGPT-3.5 and GPT-4 on the United States Medical Licensing Examination With and Without Distractions [17.813396230160095]
  This study investigates the impact of medical data mixed with small talk on the accuracy of medical advice provided by ChatGPT. We gathered small talk sentences from human participants using the Mechanical Turk platform. ChatGPT-4 seems more accurate than the earlier 3.5 version, and it appears that small talk does not impair its capability to provide medical recommendations.
  arXiv  Detail & Related papers  (2023-09-12T05:54:45Z)
• A Generic Deep Learning Based Cough Analysis System from Clinically Validated Samples for Point-of-Need Covid-19 Test and Severity Levels [85.41238731489939]
  We seek to evaluate the detection performance of a rapid primary screening tool of Covid-19 based on the cough sound from 8,380 clinically validated samples. Our proposed generic method is an algorithm based on Empirical Mode Decomposition (EMD) with subsequent classification based on a tensor of audio features. Two different versions of DeepCough based on the number of tensor dimensions, i.e. DeepCough2D and DeepCough3D, have been investigated.
  arXiv  Detail & Related papers  (2021-11-10T19:39:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.