Accelerating Clinical Evidence Synthesis with Large Language Models

• URL: http://arxiv.org/abs/2406.17755v2
• Date: Mon, 28 Oct 2024 22:41:19 GMT
• Title: Accelerating Clinical Evidence Synthesis with Large Language Models
• Authors: Zifeng Wang, Lang Cao, Benjamin Danek, Qiao Jin, Zhiyong Lu, Jimeng Sun,
• Abstract summary: We introduce TrialMind, a generative artificial intelligence pipeline for facilitating human-AI collaboration. TrialMind excels across study search, screening, and data extraction tasks. Human experts favored TrialMind's outputs over GPT-4's in 62.5% to 100% of cases.
• Score: 28.002870749019035
• License:
• Abstract: Synthesizing clinical evidence largely relies on systematic reviews of clinical trials and retrospective analyses from medical literature. However, the rapid expansion of publications presents challenges in efficiently identifying, summarizing, and updating clinical evidence. Here, we introduce TrialMind, a generative artificial intelligence (AI) pipeline for facilitating human-AI collaboration in three crucial tasks for evidence synthesis: study search, screening, and data extraction. To assess its performance, we chose published systematic reviews to build the benchmark dataset, named TrialReviewBench, which contains 100 systematic reviews and the associated 2,220 clinical studies. Our results show that TrialMind excels across all three tasks. In study search, it generates diverse and comprehensive search queries to achieve high recall rates (Ours 0.711-0.834 v.s. Human baseline 0.138-0.232). For study screening, TrialMind surpasses traditional embedding-based methods by 30% to 160%. In data extraction, it outperforms a GPT-4 baseline by 29.6% to 61.5%. We further conducted user studies to confirm its practical utility. Compared to manual efforts, human-AI collaboration using TrialMind yielded a 71.4% recall lift and 44.2% time savings in study screening and a 23.5% accuracy lift and 63.4% time savings in data extraction. Additionally, when comparing synthesized clinical evidence presented in forest plots, medical experts favored TrialMind's outputs over GPT-4's outputs in 62.5% to 100% of cases. These findings show the promise of LLM-based approaches like TrialMind to accelerate clinical evidence synthesis via streamlining study search, screening, and data extraction from medical literature, with exceptional performance improvement when working with human experts.

Related papers

• End-To-End Clinical Trial Matching with Large Language Models [0.6151041580858937]
  We present an end-to-end pipeline for clinical trial matching using Large Language Models (LLMs) Our approach identifies relevant candidate trials in 93.3% of cases and achieves a preliminary accuracy of 88.0%. Our fully end-to-end pipeline can operate autonomously or with human supervision and is not restricted to oncology.
  arXiv  Detail & Related papers  (2024-07-18T12:36:26Z)
• TrialBench: Multi-Modal Artificial Intelligence-Ready Clinical Trial Datasets [57.067409211231244]
  This paper presents meticulously curated AIready datasets covering multi-modal data (e.g., drug molecule, disease code, text, categorical/numerical features) and 8 crucial prediction challenges in clinical trial design. We provide basic validation methods for each task to ensure the datasets' usability and reliability. We anticipate that the availability of such open-access datasets will catalyze the development of advanced AI approaches for clinical trial design.
  arXiv  Detail & Related papers  (2024-06-30T09:13:10Z)
• De-identification of clinical free text using natural language processing: A systematic review of current approaches [48.343430343213896]
  Natural language processing has repeatedly demonstrated its feasibility in automating the de-identification process. Our study aims to provide systematic evidence on how the de-identification of clinical free text has evolved in the last thirteen years.
  arXiv  Detail & Related papers  (2023-11-28T13:20:41Z)
• Investigating Deep-Learning NLP for Automating the Extraction of Oncology Efficacy Endpoints from Scientific Literature [0.0]
  We have developed and optimised a framework to extract efficacy endpoints from text in scientific papers. Our machine learning model predicts 25 classes associated with efficacy endpoints and leads to high F1 scores.
  arXiv  Detail & Related papers  (2023-11-03T14:01:54Z)
• Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding [31.884600238089405]
  We present Clinical Camel, an open large language model (LLM) explicitly tailored for clinical research. Fine-tuned from LLaMA-2 using QLoRA, Clinical Camel achieves state-of-the-art performance across medical benchmarks among openly available medical LLMs.
  arXiv  Detail & Related papers  (2023-05-19T23:07:09Z)
• SPOT: Sequential Predictive Modeling of Clinical Trial Outcome with Meta-Learning [67.8195828626489]
  Clinical trials are essential to drug development but time-consuming, costly, and prone to failure. We propose Sequential Predictive mOdeling of clinical Trial outcome (SPOT) that first identifies trial topics to cluster the multi-sourced trial data into relevant trial topics. With the consideration of each trial sequence as a task, it uses a meta-learning strategy to achieve a point where the model can rapidly adapt to new tasks with minimal updates.
  arXiv  Detail & Related papers  (2023-04-07T23:04:27Z)
• Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching [49.78442796596806]
  We propose an innovative privacy-aware data augmentation approach for patient-trial matching (LLM-PTM) Our experiments demonstrate a 7.32% average improvement in performance using the proposed LLM-PTM method, and the generalizability to new data is improved by 12.12%.
  arXiv  Detail & Related papers  (2023-03-24T03:14:00Z)
• ANOVA-based Automatic Attribute Selection and a Predictive Model for Heart Disease Prognosis [8.258177970935085]
  This work proposes an information fusion technique that combines key attributes of a person through analysis of variance (ANOVA) and domain experts' knowledge. The proposed approach can achieve a competitive mean average accuracy (mAA) of 99.2% and a mean average AUC of 97.9%.
  arXiv  Detail & Related papers  (2022-07-30T19:29:18Z)
• A Systematic Review of Natural Language Processing Applied to Radiology Reports [3.600747505433814]
  This study systematically assesses recent literature in NLP applied to radiology reports. Our analysis is based on 21 variables including radiology characteristics, NLP methodology, performance, study, and clinical application characteristics.
  arXiv  Detail & Related papers  (2021-02-18T18:54:41Z)
• HINT: Hierarchical Interaction Network for Trial Outcome Prediction Leveraging Web Data [56.53715632642495]
  Clinical trials face uncertain outcomes due to issues with efficacy, safety, or problems with patient recruitment. In this paper, we propose Hierarchical INteraction Network (HINT) for more general, clinical trial outcome predictions.
  arXiv  Detail & Related papers  (2021-02-08T15:09:07Z)
• CovidDeep: SARS-CoV-2/COVID-19 Test Based on Wearable Medical Sensors and Efficient Neural Networks [51.589769497681175]
  The novel coronavirus (SARS-CoV-2) has led to a pandemic. The current testing regime based on Reverse Transcription-Polymerase Chain Reaction for SARS-CoV-2 has been unable to keep up with testing demands. We propose a framework called CovidDeep that combines efficient DNNs with commercially available WMSs for pervasive testing of the virus.
  arXiv  Detail & Related papers  (2020-07-20T21:47:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.