VoxelPrompt: A Vision-Language Agent for Grounded Medical Image Analysis

• URL: http://arxiv.org/abs/2410.08397v1
• Date: Thu, 10 Oct 2024 22:11:43 GMT
• Title: VoxelPrompt: A Vision-Language Agent for Grounded Medical Image Analysis
• Authors: Andrew Hoopes, Victor Ion Butoi, John V. Guttag, Adrian V. Dalca,
• Abstract summary: VoxelPrompt tackles diverse radiological tasks through joint modeling of natural language, image volumes, and analytical metrics. We show that VoxelPrompt can delineate hundreds of anatomical and pathological features, measure many complex morphological properties, and perform open-language analysis of lesion characteristics.
• Score: 9.937830036053871
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present VoxelPrompt, an agent-driven vision-language framework that tackles diverse radiological tasks through joint modeling of natural language, image volumes, and analytical metrics. VoxelPrompt is multi-modal and versatile, leveraging the flexibility of language interaction while providing quantitatively grounded image analysis. Given a variable number of 3D medical volumes, such as MRI and CT scans, VoxelPrompt employs a language agent that iteratively predicts executable instructions to solve a task specified by an input prompt. These instructions communicate with a vision network to encode image features and generate volumetric outputs (e.g., segmentations). VoxelPrompt interprets the results of intermediate instructions and plans further actions to compute discrete measures (e.g., tumor growth across a series of scans) and present relevant outputs to the user. We evaluate this framework in a sandbox of diverse neuroimaging tasks, and we show that the single VoxelPrompt model can delineate hundreds of anatomical and pathological features, measure many complex morphological properties, and perform open-language analysis of lesion characteristics. VoxelPrompt carries out these objectives with accuracy similar to that of fine-tuned, single-task models for segmentation and visual question-answering, while facilitating a much larger range of tasks. Therefore, by supporting accurate image processing with language interaction, VoxelPrompt provides comprehensive utility for numerous imaging tasks that traditionally require specialized models to address.

Related papers

• Computed Tomography Visual Question Answering with Cross-modal Feature Graphing [16.269682136158004]
  Visual question answering (VQA) in medical imaging aims to support clinical diagnosis by automatically interpreting complex imaging data in response to natural language queries.<n>Existing studies typically rely on distinct visual and textual encoders to independently extract features from medical images and clinical questions, which are subsequently combined to generate answers.<n>We propose a novel large language model (LLM)-based framework enhanced by a graph representation of salient features.
  arXiv  Detail & Related papers  (2025-07-06T10:37:16Z)
• MedSeg-R: Reasoning Segmentation in Medical Images with Multimodal Large Language Models [48.24824129683951]
  We introduce medical image reasoning segmentation, a novel task that aims to generate segmentation masks based on complex and implicit medical instructions.<n>To address this, we propose MedSeg-R, an end-to-end framework that leverages the reasoning abilities of MLLMs to interpret clinical questions.<n>It is built on two core components: 1) a global context understanding module that interprets images and comprehends complex medical instructions to generate multi-modal intermediate tokens, and 2) a pixel-level grounding module that decodes these tokens to produce precise segmentation masks.
  arXiv  Detail & Related papers  (2025-06-12T08:13:38Z)
• MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations [13.991376926757036]
  We propose MedUnifier, a unified Vision-Language Pre-Training framework tailored for medical data. MedUnifier seamlessly integrates text-grounded image generation capabilities with multi-modal learning strategies. Our approach employs visual vector quantization, which not only facilitates a more cohesive learning strategy for cross-modal understanding but also enhances multi-modal generation quality.
  arXiv  Detail & Related papers  (2025-03-02T21:09:32Z)
• Autoregressive Sequence Modeling for 3D Medical Image Representation [48.706230961589924]
  We introduce a pioneering method for learning 3D medical image representations through an autoregressive sequence pre-training framework. Our approach various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence.
  arXiv  Detail & Related papers  (2024-09-13T10:19:10Z)
• FluoroSAM: A Language-promptable Foundation Model for Flexible X-ray Image Segmentation [11.55858990545478]
  FluoroSAM is a language-promptable variant of the Segment Anything Model.<n>It is capable of segmenting myriad anatomical structures and tools based on natural language prompts.<n>We show how FluoroSAM is a key enabler for rich human-machine interaction in the X-ray image acquisition and analysis context.
  arXiv  Detail & Related papers  (2024-03-12T20:11:38Z)
• MedXChat: A Unified Multimodal Large Language Model Framework towards CXRs Understanding and Generation [28.497591315598402]
  Multimodal Large Language Models (MLLMs) have shown success in various general image processing tasks. This study investigates the potential of MLLMs in improving the understanding and generation of Chest X-Rays (CXRs)
  arXiv  Detail & Related papers  (2023-12-04T06:40:12Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models [0.8878802873945023]
  This study introduces the first systematic study on transferring Vision-Language Models to 2D medical images. Although VLSMs show competitive performance compared to image-only models for segmentation, not all VLSMs utilize the additional information from language prompts.
  arXiv  Detail & Related papers  (2023-08-15T11:28:21Z)
• XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models [60.437091462613544]
  We introduce XrayGPT, a novel conversational medical vision-language model. It can analyze and answer open-ended questions about chest radiographs. We generate 217k interactive and high-quality summaries from free-text radiology reports.
  arXiv  Detail & Related papers  (2023-06-13T17:59:59Z)
• PaLI-X: On Scaling up a Multilingual Vision and Language Model [166.9837904115951]
  We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model. Our model achieves new levels of performance on a wide-range of varied and complex tasks. We observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.
  arXiv  Detail & Related papers  (2023-05-29T18:58:38Z)
• Kartezio: Evolutionary Design of Explainable Pipelines for Biomedical Image Analysis [0.0]
  We introduce Kartezio, a computational strategy that generates transparent and easily interpretable image processing pipelines. The pipelines thus generated exhibit comparable precision to state-of-the-art Deep Learning approaches on instance segmentation tasks. We also deployed Kartezio to solve semantic and instance segmentation problems in four real-world Use Cases.
  arXiv  Detail & Related papers  (2023-02-28T17:02:35Z)
• Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts [63.84720380390935]
  There exist two typical types, textiti.e., the fusion-encoder type and the dual-encoder type, depending on whether a heavy fusion module is used. We propose an effective yet straightforward scheme named PTUnifier to unify the two types. We first unify the input format by introducing visual and textual prompts, which serve as a feature bank that stores the most representative images/texts.
  arXiv  Detail & Related papers  (2023-02-17T15:43:42Z)
• Universal Multimodal Representation for Language Understanding [110.98786673598015]
  This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively.
  arXiv  Detail & Related papers  (2023-01-09T13:54:11Z)
• Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training [62.215025958347105]
  We propose a self-supervised learning paradigm with multi-modal masked autoencoders. We learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts.
  arXiv  Detail & Related papers  (2022-09-15T07:26:43Z)
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training [5.119201893752376]
  We propose Medical Vision Language Learner (MedViLL) which adopts a Transformer-based architecture combined with a novel multimodal attention masking scheme. We empirically demonstrate the superior downstream task performance of MedViLL against various baselines including task-specific architectures.
  arXiv  Detail & Related papers  (2021-05-24T15:14:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.