LLM-driven Multimodal Target Volume Contouring in Radiation Oncology

• URL: http://arxiv.org/abs/2311.01908v4
• Date: Thu, 24 Oct 2024 18:10:26 GMT
• Title: LLM-driven Multimodal Target Volume Contouring in Radiation Oncology
• Authors: Yujin Oh, Sangjoon Park, Hwa Kyung Byun, Yeona Cho, Ik Jae Lee, Jin Sung Kim, Jong Chul Ye,
• Abstract summary: Large language models (LLMs) can facilitate the integration of the textural information and images. We present a novel LLM-driven multimodal AI, namely LLMSeg, that is applicable to the challenging task of target volume contouring for radiation therapy. We demonstrate that the proposed model exhibits markedly improved performance compared to conventional unimodal AI models.
• Score: 46.23891509553877
• License:
• Abstract: Target volume contouring for radiation therapy is considered significantly more challenging than the normal organ segmentation tasks as it necessitates the utilization of both image and text-based clinical information. Inspired by the recent advancement of large language models (LLMs) that can facilitate the integration of the textural information and images, here we present a novel LLM-driven multimodal AI, namely LLMSeg, that utilizes the clinical text information and is applicable to the challenging task of target volume contouring for radiation therapy, and validate it within the context of breast cancer radiation therapy target volume contouring. Using external validation and data-insufficient environments, which attributes highly conducive to real-world applications, we demonstrate that the proposed model exhibits markedly improved performance compared to conventional unimodal AI models, particularly exhibiting robust generalization performance and data efficiency. To our best knowledge, this is the first LLM-driven multimodal AI model that integrates the clinical text information into target volume delineation for radiation oncology.

Related papers

• RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction [20.974460332254544]
  RespLLM is a novel framework that unifies text and audio representations for respiratory health prediction. Our work lays the foundation for multimodal models that can perceive, listen, and understand heterogeneous data.
  arXiv  Detail & Related papers  (2024-10-07T17:06:11Z)
• The Era of Foundation Models in Medical Imaging is Approaching : A Scoping Review of the Clinical Value of Large-Scale Generative AI Applications in Radiology [0.0]
  Social problems stemming from the shortage of radiologists are intensifying, and artificial intelligence is being highlighted as a potential solution. Recently emerging large-scale generative AI has expanded from large language models (LLMs) to multi-modal models. This scoping review systematically organizes existing literature on the clinical value of large-scale generative AI applications.
  arXiv  Detail & Related papers  (2024-09-03T00:48:50Z)
• MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis [6.30440420617113]
  We introduce MedTsLLM, a general multimodal large language model (LLM) framework that integrates time series data and rich contextual information in the form of text to analyze physiological signals. We perform three tasks with clinical relevance: semantic segmentation, boundary detection, and anomaly detection in time series. Our model outperforms state-of-the-art baselines, including deep learning models, other LLMs, and clinical methods across multiple medical domains.
  arXiv  Detail & Related papers  (2024-08-14T18:57:05Z)
• DALL-M: Context-Aware Clinical Data Augmentation with LLMs [13.827368628263997]
  Radiologists often find chest X-rays insufficient for diagnosing underlying diseases. We present a novel framework to enhance the clinical context through augmentation techniques with clinical data. We introduce a pioneering approach to clinical data augmentation that employs large language models to generate patient contextual synthetic data.
  arXiv  Detail & Related papers  (2024-07-11T07:01:50Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• Large Language Model Distilling Medication Recommendation Model [61.89754499292561]
  We harness the powerful semantic comprehension and input-agnostic characteristics of Large Language Models (LLMs) Our research aims to transform existing medication recommendation methodologies using LLMs. To mitigate this, we have developed a feature-level knowledge distillation technique, which transfers the LLM's proficiency to a more compact model.
  arXiv  Detail & Related papers  (2024-02-05T08:25:22Z)
• End-to-End Breast Cancer Radiotherapy Planning via LMMs with Consistency Embedding [47.360760580820966]
  We present RO-LMM, a comprehensive large multimodal model (LMM) tailored for the field of radiation oncology. This model effectively manages a series of tasks within the clinical workflow, including clinical context summarization, radiation treatment plan suggestion, and plan-guided target volume segmentation. We present a novel Consistency Embedding Fine-Tuning (CEFTune) technique, which boosts LMM's robustness to noisy inputs while preserving the consistency of handling clean inputs.
  arXiv  Detail & Related papers  (2023-11-27T14:49:06Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT [80.33783969507458]
  The 'Impression' section of a radiology report is a critical basis for communication between radiologists and other physicians. Recent studies have achieved promising results in automatic impression generation using large-scale medical text data. These models often require substantial amounts of medical text data and have poor generalization performance.
  arXiv  Detail & Related papers  (2023-04-17T17:13:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.