DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models

• URL: http://arxiv.org/abs/2408.01933v2
• Date: Tue, 6 Aug 2024 04:28:01 GMT
• Title: DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models
• Authors: Bowen Wang, Jiuyang Chang, Yiming Qian, Guoxin Chen, Junhao Chen, Zhouqiang Jiang, Jiahao Zhang, Yuta Nakashima, Hajime Nagahara,
• Abstract summary: We aim at evaluating the reasoning ability and interpretability of large language models (LLMs) compared to human doctors. The diagnostic reasoning dataset for clinical notes (DiReCT) contains 511 clinical notes, each meticulously annotated by physicians.
• Score: 32.85606857702375
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have recently showcased remarkable capabilities, spanning a wide range of tasks and applications, including those in the medical domain. Models like GPT-4 excel in medical question answering but may face challenges in the lack of interpretability when handling complex tasks in real clinical settings. We thus introduce the diagnostic reasoning dataset for clinical notes (DiReCT), aiming at evaluating the reasoning ability and interpretability of LLMs compared to human doctors. It contains 511 clinical notes, each meticulously annotated by physicians, detailing the diagnostic reasoning process from observations in a clinical note to the final diagnosis. Additionally, a diagnostic knowledge graph is provided to offer essential knowledge for reasoning, which may not be covered in the training data of existing LLMs. Evaluations of leading LLMs on DiReCT bring out a significant gap between their reasoning ability and that of human doctors, highlighting the critical need for models that can reason effectively in real-world clinical scenarios.

Related papers

• CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios [50.032101237019205]
  CliMedBench is a comprehensive benchmark with 14 expert-guided core clinical scenarios. The reliability of this benchmark has been confirmed in several ways.
  arXiv  Detail & Related papers  (2024-10-04T15:15:36Z)
• RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment [54.91736546490813]
  We introduce the RuleAlign framework, designed to align Large Language Models with specific diagnostic rules. We develop a medical dialogue dataset comprising rule-based communications between patients and physicians. Experimental results demonstrate the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2024-08-22T17:44:40Z)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI [67.09501109871351]
  Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals. GMAI-MMBench is the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  arXiv  Detail & Related papers  (2024-08-06T17:59:21Z)
• SemioLLM: Assessing Large Language Models for Semiological Analysis in Epilepsy Research [45.2233252981348]
  Large Language Models have shown promising results in their ability to encode general medical knowledge. We test the ability of state-of-the-art LLMs to leverage their internal knowledge and reasoning for epilepsy diagnosis.
  arXiv  Detail & Related papers  (2024-07-03T11:02:12Z)
• CliBench: A Multifaceted and Multigranular Evaluation of Large Language Models for Clinical Decision Making [16.310913127940857]
  We introduce CliBench, a novel benchmark developed from the MIMIC IV dataset. This benchmark offers a comprehensive and realistic assessment of LLMs' capabilities in clinical diagnosis. We conduct a zero-shot evaluation of leading LLMs to assess their proficiency in clinical decision-making.
  arXiv  Detail & Related papers  (2024-06-14T11:10:17Z)
• Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding [53.629132242389716]
  Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions. VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information. We propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge.
  arXiv  Detail & Related papers  (2024-05-29T23:19:28Z)
• Large Language Models are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales [15.362903610463285]
  We present a "reasoning-aware" diagnosis framework that rationalizes the diagnostic process via prompt-based learning. We propose a novel set of criteria for evaluating machine-generated rationales' potential for real-world clinical settings.
  arXiv  Detail & Related papers  (2023-12-12T16:14:45Z)
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization [8.456700096020601]
  Large language models (LLMs) have shown promise in natural language processing (NLP), but their effectiveness on a diverse range of clinical summarization tasks remains unproven. In this study, we apply adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks. A clinical reader study with ten physicians evaluates summary, completeness, correctness, and conciseness; in a majority of cases, summaries from our best adapted LLMs are either equivalent (45%) or superior (36%) compared to summaries from medical experts.
  arXiv  Detail & Related papers  (2023-09-14T05:15:01Z)
• Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine [4.773117448586697]
  We develop novel diagnostic reasoning prompts to study whether large language models (LLMs) can perform clinical reasoning to accurately form a diagnosis. We find GPT4 can be prompted to mimic the common clinical reasoning processes of clinicians without sacrificing diagnostic accuracy.
  arXiv  Detail & Related papers  (2023-08-13T19:04:07Z)
• VBridge: Connecting the Dots Between Features, Explanations, and Data for Healthcare Models [85.4333256782337]
  VBridge is a visual analytics tool that seamlessly incorporates machine learning explanations into clinicians' decision-making workflow. We identified three key challenges, including clinicians' unfamiliarity with ML features, lack of contextual information, and the need for cohort-level evidence. We demonstrated the effectiveness of VBridge through two case studies and expert interviews with four clinicians.
  arXiv  Detail & Related papers  (2021-08-04T17:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.