Automated radiotherapy treatment planning guided by GPT-4Vision

• URL: http://arxiv.org/abs/2406.15609v2
• Date: Mon, 1 Jul 2024 21:45:44 GMT
• Title: Automated radiotherapy treatment planning guided by GPT-4Vision
• Authors: Sheng Liu, Oscar Pastor-Serrano, Yizheng Chen, Matthew Gopaulchan, Weixing Liang, Mark Buyyounouski, Erqi Pollom, Quynh-Thu Le, Michael Gensheimer, Peng Dong, Yong Yang, James Zou, Lei Xing,
• Abstract summary: This study introduces GPT-RadPlan, a fully automated treatment planning framework. GPT-RadPlan harnesses prior radiation oncology knowledge encoded in multi-modal large language models, such as GPT-4Vision (GPT-4V) from OpenAI. GPT-RadPlan is integrated into our in-house inverse treatment planning system through an API.
• Score: 27.56613357226252
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Radiotherapy treatment planning is a time-consuming and potentially subjective process that requires the iterative adjustment of model parameters to balance multiple conflicting objectives. Recent advancements in large foundation models offer promising avenues for addressing the challenges in planning and clinical decision-making. This study introduces GPT-RadPlan, a fully automated treatment planning framework that harnesses prior radiation oncology knowledge encoded in multi-modal large language models, such as GPT-4Vision (GPT-4V) from OpenAI. GPT-RadPlan is made aware of planning protocols as context and acts as an expert human planner, capable of guiding a treatment planning process. Via in-context learning, we incorporate clinical protocols for various disease sites as prompts to enable GPT-4V to acquire treatment planning domain knowledge. The resulting GPT-RadPlan agent is integrated into our in-house inverse treatment planning system through an API. The efficacy of the automated planning system is showcased using multiple prostate and head & neck cancer cases, where we compared GPT-RadPlan results to clinical plans. In all cases, GPT-RadPlan either outperformed or matched the clinical plans, demonstrating superior target coverage and organ-at-risk sparing. Consistently satisfying the dosimetric objectives in the clinical protocol, GPT-RadPlan represents the first multimodal large language model agent that mimics the behaviors of human planners in radiation oncology clinics, achieving remarkable results in automating the treatment planning process without the need for additional training.

Related papers

• Automating proton PBS treatment planning for head and neck cancers using policy gradient-based deep reinforcement learning [0.7519872646378836]
  We propose an automatic treatment planning model using the proximal policy optimization (PPO) algorithm and a dose distribution-based reward function. A set of empirical rules is used to create auxiliary planning structures from target volumes and organs-at-risk. A decision-making policy network trained using PPO is developed to iteratively adjust the involved planning objective parameters in a continuous action space.
  arXiv  Detail & Related papers  (2024-09-17T22:01:56Z)
• ARANet: Attention-based Residual Adversarial Network with Deep Supervision for Radiotherapy Dose Prediction of Cervical Cancer [5.737832138199829]
  We propose an end-to-end Attentionbased Residual Adversarial Network with deep supervision, namely ARANet, to automatically predict the 3D dose distribution of cervical cancer. Our proposed method is validated on an in-house dataset including 54 cervical cancer patients, and experimental results have demonstrated its obvious superiority compared to other state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-26T02:26:09Z)
• Brain Tumor Segmentation (BraTS) Challenge 2024: Meningioma Radiotherapy Planning Automated Segmentation [47.119513326344126]
  The BraTS-MEN-RT challenge aims to advance automated segmentation algorithms using the largest known multi-institutional dataset of radiotherapy planning brain MRIs. Each case includes a defaced 3D post-contrast T1-weighted radiotherapy planning MRI in its native acquisition space. Target volume annotations adhere to established radiotherapy planning protocols.
  arXiv  Detail & Related papers  (2024-05-28T17:25:43Z)
• Large-Language-Model Empowered Dose Volume Histogram Prediction for Intensity Modulated Radiotherapy [11.055104826451126]
  We propose a pipeline to convert unstructured images to a structured graph consisting of image-patch nodes and dose nodes. A novel Dose Graph Neural Network (DoseGNN) model is developed for predicting Dose-Volume histograms (DVHs) from the structured graph. In this study, we introduced an online human-AI collaboration system as a practical implementation of the concept proposed for the automation of intensity-modulated radiotherapy (IMRT) planning.
  arXiv  Detail & Related papers  (2024-02-11T11:24:09Z)
• Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty [56.30846158280031]
  Task planning for embodied AI has been one of the most challenging problems. We propose a task-agnostic method named 'planning as in-painting' The proposed framework achieves promising performances in various embodied AI tasks.
  arXiv  Detail & Related papers  (2023-12-02T10:07:17Z)
• EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought [95.37585041654535]
  Embodied AI is capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments. In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI. Experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering.
  arXiv  Detail & Related papers  (2023-05-24T11:04:30Z)
• PlanT: Explainable Planning Transformers via Object-Level Representations [64.93938686101309]
  PlanT is a novel approach for planning in the context of self-driving. PlanT is based on imitation learning with a compact object-level input representation. Our results indicate that PlanT can focus on the most relevant object in the scene, even when this object is geometrically distant.
  arXiv  Detail & Related papers  (2022-10-25T17:59:46Z)
• OpenKBP-Opt: An international and reproducible evaluation of 76 knowledge-based planning pipelines [48.547200649819615]
  We establish an open framework for developing plan optimization models for knowledge-based planning (KBP) in radiotherapy. Our framework includes reference plans for 100 patients with head-and-neck cancer and high-quality dose predictions from 19 KBP models.
  arXiv  Detail & Related papers  (2022-02-16T19:18:42Z)
• A feasibility study of a hyperparameter tuning approach to automated inverse planning in radiotherapy [68.8204255655161]
  The purpose of this study is to automate the inverse planning process to reduce active planning time while maintaining plan quality. We investigated the impact of the choice of dose parameters, random and Bayesian search methods, and utility function form on planning time and plan quality. Using 100 samples was found to produce satisfactory plan quality, and the average planning time was 2.3 hours.
  arXiv  Detail & Related papers  (2021-05-14T18:37:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.