Towards Accurate Differential Diagnosis with Large Language Models
- URL: http://arxiv.org/abs/2312.00164v1
- Date: Thu, 30 Nov 2023 19:55:51 GMT
- Title: Towards Accurate Differential Diagnosis with Large Language Models
- Authors: Daniel McDuff and Mike Schaekermann and Tao Tu and Anil Palepu and Amy
Wang and Jake Garrison and Karan Singhal and Yash Sharma and Shekoofeh Azizi
and Kavita Kulkarni and Le Hou and Yong Cheng and Yun Liu and S Sara Mahdavi
and Sushant Prakash and Anupam Pathak and Christopher Semturs and Shwetak
Patel and Dale R Webster and Ewa Dominowska and Juraj Gottweis and Joelle
Barral and Katherine Chou and Greg S Corrado and Yossi Matias and Jake
Sunshine and Alan Karthikesalingam and Vivek Natarajan
- Abstract summary: Interactive interfaces powered by Large Language Models (LLMs) present new opportunities to both assist and automate aspects of differential diagnosis.
20 clinicians evaluated 302 challenging, real-world medical cases sourced from the New England Journal of Medicine.
Our study suggests that our LLM has potential to improve clinicians' diagnostic reasoning and accuracy in challenging cases.
- Score: 37.48155380562073
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: An accurate differential diagnosis (DDx) is a cornerstone of medical care,
often reached through an iterative process of interpretation that combines
clinical history, physical examination, investigations and procedures.
Interactive interfaces powered by Large Language Models (LLMs) present new
opportunities to both assist and automate aspects of this process. In this
study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
evaluated 302 challenging, real-world medical cases sourced from the New
England Journal of Medicine (NEJM) case reports. Each case report was read by
two clinicians, who were randomized to one of two assistive conditions: either
assistance from search engines and standard medical resources, or LLM
assistance in addition to these tools. All clinicians provided a baseline,
unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
exhibited standalone performance that exceeded that of unassisted clinicians
(top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
arms, the DDx quality score was higher for clinicians assisted by our LLM
(top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
(McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
differential lists than those without its assistance. Our study suggests that
our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
accuracy in challenging cases, meriting further real-world evaluation for its
ability to empower physicians and widen patients' access to specialist-level
expertise.
Related papers
- CliBench: Multifaceted Evaluation of Large Language Models in Clinical Decisions on Diagnoses, Procedures, Lab Tests Orders and Prescriptions [16.310913127940857]
We introduce CliBench, a novel benchmark developed from the MIMIC IV dataset.
This benchmark offers a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis.
We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making.
arXiv Detail & Related papers (2024-06-14T11:10:17Z) - Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments [7.219693607724636]
We aim to tackle this shortage by integrating a customized large language model (LLM) into the workflow.
We collect 411 clinician-administered diagnostic interviews and devise a novel approach to obtain high-quality data.
We build a comprehensive framework to automate PTSD diagnostic assessments based on interview contents.
arXiv Detail & Related papers (2024-05-18T05:04:18Z) - AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs.
This setup allows for realistic assessments of LLMs in clinical scenarios.
We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
arXiv Detail & Related papers (2024-02-15T06:46:48Z) - Zero-Shot Clinical Trial Patient Matching with LLMs [40.31971412825736]
Large language models (LLMs) offer a promising solution to automated screening.
We design an LLM-based system which, given a patient's medical history as unstructured clinical text, evaluates whether that patient meets a set of inclusion criteria.
Our system achieves state-of-the-art scores on the n2c2 2018 cohort selection benchmark.
arXiv Detail & Related papers (2024-02-05T00:06:08Z) - Towards Conversational Diagnostic AI [32.84876349808714]
We introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue.
AMIE uses a self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions.
AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors.
arXiv Detail & Related papers (2024-01-11T04:25:06Z) - Deciphering Diagnoses: How Large Language Models Explanations Influence
Clinical Decision Making [0.0]
Large Language Models (LLMs) are emerging as a promising tool to generate plain-text explanations for medical decisions.
This study explores the effectiveness and reliability of LLMs in generating explanations for diagnoses based on patient complaints.
arXiv Detail & Related papers (2023-10-03T00:08:23Z) - MedAlign: A Clinician-Generated Dataset for Instruction Following with
Electronic Medical Records [60.35217378132709]
Large language models (LLMs) can follow natural language instructions with human-level fluency.
evaluating LLMs on realistic text generation tasks for healthcare remains challenging.
We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
arXiv Detail & Related papers (2023-08-27T12:24:39Z) - Self-Verification Improves Few-Shot Clinical Information Extraction [73.6905567014859]
Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning.
They still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health.
Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs.
arXiv Detail & Related papers (2023-05-30T22:05:11Z) - SPeC: A Soft Prompt-Based Calibration on Performance Variability of
Large Language Model in Clinical Notes Summarization [50.01382938451978]
We introduce a model-agnostic pipeline that employs soft prompts to diminish variance while preserving the advantages of prompt-based summarization.
Experimental findings indicate that our method not only bolsters performance but also effectively curbs variance for various language models.
arXiv Detail & Related papers (2023-03-23T04:47:46Z) - Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in
Artificial Intelligence [79.038671794961]
We launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution.
Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK.
arXiv Detail & Related papers (2021-11-18T00:43:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.