Towards Conversational Diagnostic AI
- URL: http://arxiv.org/abs/2401.05654v1
- Date: Thu, 11 Jan 2024 04:25:06 GMT
- Title: Towards Conversational Diagnostic AI
- Authors: Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg,
Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, Shekoofeh
Azizi, Karan Singhal, Yong Cheng, Le Hou, Albert Webson, Kavita Kulkarni, S
Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine
Chou, Greg S Corrado, Yossi Matias, Alan Karthikesalingam and Vivek Natarajan
- Abstract summary: We introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue.
AMIE uses a self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions.
AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors.
- Score: 32.84876349808714
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: At the heart of medicine lies the physician-patient dialogue, where skillful
history-taking paves the way for accurate diagnosis, effective management, and
enduring trust. Artificial Intelligence (AI) systems capable of diagnostic
dialogue could increase accessibility, consistency, and quality of care.
However, approximating clinicians' expertise is an outstanding grand challenge.
Here, we introduce AMIE (Articulate Medical Intelligence Explorer), a Large
Language Model (LLM) based AI system optimized for diagnostic dialogue.
AMIE uses a novel self-play based simulated environment with automated
feedback mechanisms for scaling learning across diverse disease conditions,
specialties, and contexts. We designed a framework for evaluating
clinically-meaningful axes of performance including history-taking, diagnostic
accuracy, management reasoning, communication skills, and empathy. We compared
AMIE's performance to that of primary care physicians (PCPs) in a randomized,
double-blind crossover study of text-based consultations with validated patient
actors in the style of an Objective Structured Clinical Examination (OSCE). The
study included 149 case scenarios from clinical providers in Canada, the UK,
and India, 20 PCPs for comparison with AMIE, and evaluations by specialist
physicians and patient actors. AMIE demonstrated greater diagnostic accuracy
and superior performance on 28 of 32 axes according to specialist physicians
and 24 of 26 axes according to patient actors. Our research has several
limitations and should be interpreted with appropriate caution. Clinicians were
limited to unfamiliar synchronous text-chat which permits large-scale
LLM-patient interactions but is not representative of usual clinical practice.
While further research is required before AMIE could be translated to
real-world settings, the results represent a milestone towards conversational
diagnostic AI.
Related papers
- Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding [53.629132242389716]
Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions.
VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information.
We propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge.
arXiv Detail & Related papers (2024-05-29T23:19:28Z) - AgentClinic: a multimodal agent benchmark to evaluate AI in simulated clinical environments [2.567146936147657]
We present AgentClinic: a benchmark to evaluate large language models (LLMs) in simulated clinical environments.
In our benchmark, the doctor agent must uncover the patient's diagnosis through dialogue and active data collection.
We find that introducing bias leads to large reductions in diagnostic accuracy of the doctor agents, as well as reduced compliance, confidence, and follow-up willingness in patient agents.
arXiv Detail & Related papers (2024-05-13T17:38:53Z) - Conversational Disease Diagnosis via External Planner-Controlled Large Language Models [18.93345199841588]
This study presents a LLM-based diagnostic system that enhances planning capabilities by emulating doctors.
By utilizing real patient electronic medical record data, we constructed simulated dialogues between virtual patients and doctors.
arXiv Detail & Related papers (2024-04-04T06:16:35Z) - AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs.
This setup allows for realistic assessments of LLMs in clinical scenarios.
We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
arXiv Detail & Related papers (2024-02-15T06:46:48Z) - Beyond Direct Diagnosis: LLM-based Multi-Specialist Agent Consultation
for Automatic Diagnosis [30.943705201552643]
We propose a framework to model the diagnosis process in the real world by adaptively fusing probability distributions of agents over potential diseases.
Our approach requires significantly less parameter updating and training time, enhancing efficiency and practical utility.
arXiv Detail & Related papers (2024-01-29T12:25:30Z) - Towards Accurate Differential Diagnosis with Large Language Models [37.48155380562073]
Interactive interfaces powered by Large Language Models (LLMs) present new opportunities to both assist and automate aspects of differential diagnosis.
20 clinicians evaluated 302 challenging, real-world medical cases sourced from the New England Journal of Medicine.
Our study suggests that our LLM has potential to improve clinicians' diagnostic reasoning and accuracy in challenging cases.
arXiv Detail & Related papers (2023-11-30T19:55:51Z) - Informing clinical assessment by contextualizing post-hoc explanations
of risk prediction models in type-2 diabetes [50.8044927215346]
We consider a comorbidity risk prediction scenario and focus on contexts regarding the patients clinical state.
We employ several state-of-the-art LLMs to present contexts around risk prediction model inferences and evaluate their acceptability.
Our paper is one of the first end-to-end analyses identifying the feasibility and benefits of contextual explanations in a real-world clinical use case.
arXiv Detail & Related papers (2023-02-11T18:07:11Z) - Benchmarking Automated Clinical Language Simplification: Dataset,
Algorithm, and Evaluation [48.87254340298189]
We construct a new dataset named MedLane to support the development and evaluation of automated clinical language simplification approaches.
We propose a new model called DECLARE that follows the human annotation procedure and achieves state-of-the-art performance.
arXiv Detail & Related papers (2020-12-04T06:09:02Z) - MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware
Medical Dialogue Generation [86.38736781043109]
We build and release a large-scale high-quality Medical Dialogue dataset related to 12 types of common Gastrointestinal diseases named MedDG.
We propose two kinds of medical dialogue tasks based on MedDG dataset. One is the next entity prediction and the other is the doctor response generation.
Experimental results show that the pre-train language models and other baselines struggle on both tasks with poor performance in our dataset.
arXiv Detail & Related papers (2020-10-15T03:34:33Z) - COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching [70.08786840301435]
We propose CrOss-Modal PseudO-SiamEse network (COMPOSE) to address these challenges for patient-trial matching.
Experiment results show COMPOSE can reach 98.0% AUC on patient-criteria matching and 83.7% accuracy on patient-trial matching.
arXiv Detail & Related papers (2020-06-15T21:01:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.