Large language models improve physician accuracy but lead to false reliance

• URL: http://arxiv.org/abs/2608.00817v1
• Date: Sat, 01 Aug 2026 18:48:51 GMT
• Title: Large language models improve physician accuracy but lead to false reliance
• Abstract summary: Retrieval-augmented large language models (LLMs) promise source-linked clinical support.<n> CORA investigated how source-linked assistance affects physician decision-making.<n>Findings show that source-linked LLM assistance can improve physician accuracy while introducing a grounding-dependent safety risk.
• Score: 27.854576188002074
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Retrieval-augmented large language models (LLMs) promise source-linked clinical support, but their value depends on whether displayed evidence guides rather than distorts physician reliance. We developed CORA, an agentic retrieval-augmented LLM, to investigate how source-linked assistance affects physician decision-making. CORA maintained benchmark performance and achieved larger gains on cases published after the models' training-data cutoffs. In a study of 46 physicians, accuracy increased from 70.8% unaided to 82.6% with CORA. Supporting citations predicted correct answers (87.7% vs 65.5%), but citations created an important asymmetry: perceived support increased adoption of correct advice from 34% to 76.9% but when an incorrect LLM answer appeared citation-supported, physician resistance to it fell from 92% to 34.8%. These findings show that source-linked LLM assistance can improve physician accuracy while introducing a grounding-dependent safety risk.

Related papers

• Measuring Epistemic Resilience of LLMs Under Misleading Medical Context [47.20527783144983]
  Large language models (LLMs) now reach expert-level scores on medical licensing exams.<n>We show this assumption is fragile: when misleading context is injected into questions that LLMs originally answer correctly, they abandon the correct answer.<n>We call the ability to maintain correct judgment under adversarial context epistemic resilience, and introduce MedMisBench to measure it.
  arXiv  Detail & Related papers  (2026-06-10T16:27:26Z)
• DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Training [18.22030439716779]
  Large language models (LLMs) tuned for safety often avoid acknowledging demographic differences.<n>This identity-blindness yields incorrect responses, unnecessary refusals, or generic "equal-treatment" defaults.<n>We introduce DART (Distill--Audit-Repair Training), which distills label-conditioned reasoning from a teacher.
  arXiv  Detail & Related papers  (2026-04-18T05:28:53Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• Deliberative multi-agent large language models improve clinical reasoning in ophthalmology [6.0314253430783]
  Large language models (LLMs) show potential for ophthalmic clinical reasoning, yet individual models risk introducing harm.<n>We evaluated whether multi-agent LLM deliberative councils improve diagnostic performance and mitigate harm.<n>Councils consistently outperformed pooled individual models across all three tiers.
  arXiv  Detail & Related papers  (2026-03-22T23:36:48Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology [34.82874325860935]
  Large Language Models (LLMs) in medicine may generate responses lacking supporting evidence based on hallucinated evidence. We developed a RAG pipeline with 70,000 ophthalmology-specific documents that retrieve relevant documents to augment LLMs during inference time. We evaluated the responses including over 500 references of LLMs with and without RAG on 100 questions with 10 healthcare professionals.
  arXiv  Detail & Related papers  (2024-09-20T21:06:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.