3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology

• URL: http://arxiv.org/abs/2311.13752v1
• Date: Thu, 23 Nov 2023 00:57:35 GMT
• Title: 3D-MIR: A Benchmark and Empirical Study on 3D Medical Image Retrieval in Radiology
• Authors: Asma Ben Abacha, Alberto Santamaria-Pang, Ho Hin Lee, Jameson Merkow, Qin Cai, Surya Teja Devarakonda, Abdullah Islam, Julia Gong, Matthew P. Lungren, Thomas Lin, Noel C Codella, Ivan Tarapov
• Abstract summary: The field of 3D medical image retrieval is still emerging, lacking established evaluation benchmarks, comprehensive datasets, and thorough studies. This paper introduces a novel benchmark for 3D Medical Image Retrieval (3D-MIR) that encompasses four different anatomies imaged with computed tomography. Using this benchmark, we explore a diverse set of search strategies that use aggregated 2D slices, 3D volumes, and multi-modal embeddings from popular multi-modal foundation models as queries.
• Score: 6.851500027718433
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increasing use of medical imaging in healthcare settings presents a significant challenge due to the increasing workload for radiologists, yet it also offers opportunity for enhancing healthcare outcomes if effectively leveraged. 3D image retrieval holds potential to reduce radiologist workloads by enabling clinicians to efficiently search through diagnostically similar or otherwise relevant cases, resulting in faster and more precise diagnoses. However, the field of 3D medical image retrieval is still emerging, lacking established evaluation benchmarks, comprehensive datasets, and thorough studies. This paper attempts to bridge this gap by introducing a novel benchmark for 3D Medical Image Retrieval (3D-MIR) that encompasses four different anatomies imaged with computed tomography. Using this benchmark, we explore a diverse set of search strategies that use aggregated 2D slices, 3D volumes, and multi-modal embeddings from popular multi-modal foundation models as queries. Quantitative and qualitative assessments of each approach are provided alongside an in-depth discussion that offers insight for future research. To promote the advancement of this field, our benchmark, dataset, and code are made publicly available.

Related papers

• 3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models [51.855377054763345]
  This paper introduces 3D-CT-GPT, a Visual Question Answering (VQA)-based medical visual language model for generating radiology reports from 3D CT scans. Experiments on both public and private datasets demonstrate that 3D-CT-GPT significantly outperforms existing methods in terms of report accuracy and quality.
  arXiv  Detail & Related papers  (2024-09-28T12:31:07Z)
• Autoregressive Sequence Modeling for 3D Medical Image Representation [48.706230961589924]
  We introduce a pioneering method for learning 3D medical image representations through an autoregressive sequence pre-training framework. Our approach various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence.
  arXiv  Detail & Related papers  (2024-09-13T10:19:10Z)
• Content-Based Image Retrieval for Multi-Class Volumetric Radiology Images: A Benchmark Study [0.6249768559720122]
  We benchmark embeddings derived from pre-trained supervised models on medical images against embeddings derived from pre-trained unsupervised models on non-medical images. For volumetric image retrieval, we adopt a late interaction re-ranking method inspired by text matching.
  arXiv  Detail & Related papers  (2024-05-15T13:34:07Z)
• M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models [49.5030774873328]
  Previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. We present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs. We also introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks.
  arXiv  Detail & Related papers  (2024-03-31T06:55:12Z)
• BIMCV-R: A Landmark Dataset for 3D CT Text-Image Retrieval [44.92177279141073]
  We present a dataset of 8,069 3D CT volumes, encompassing over 2 million slices, paired with their respective radiological reports. We then craft a retrieval strategy, MedFinder, harnessing the potential of large language models. It marks our preliminary step towards developing a system capable of facilitating text-to-image, image-to-text, and keyword-based retrieval tasks.
  arXiv  Detail & Related papers  (2024-03-24T03:10:07Z)
• Generative Enhancement for 3D Medical Images [74.17066529847546]
  We propose GEM-3D, a novel generative approach to the synthesis of 3D medical images. Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask. By decomposing the 3D medical images into masks and patient prior information, GEM-3D offers a flexible yet effective solution for generating versatile 3D images.
  arXiv  Detail & Related papers  (2024-03-19T15:57:04Z)
• Medical Image Retrieval Using Pretrained Embeddings [0.6827423171182154]
  We show that medical image retrieval is feasible using pretrained networks without any additional training or fine-tuning steps. Using pretrained embeddings, we achieved a recall of 1 for various tasks at modality, body region, and organ level.
  arXiv  Detail & Related papers  (2023-11-22T17:42:33Z)
• Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data [66.9359934608229]
  This study aims to initiate the development of Radiology Foundation Model, termed as RadFM. To the best of our knowledge, this is the first large-scale, high-quality, medical visual-language dataset, with both 2D and 3D scans. We propose a new evaluation benchmark, RadBench, that comprises five tasks, including modality recognition, disease diagnosis, visual question answering, report generation and rationale diagnosis.
  arXiv  Detail & Related papers  (2023-08-04T17:00:38Z)
• Medical Image Segmentation using 3D Convolutional Neural Networks: A Review [25.864941088823343]
  Computer-aided medical image analysis plays a significant role in assisting medical practitioners for expert clinical diagnosis and deciding the optimal treatment plan. At present, convolutional neural networks (CNN) are the preferred choice for medical image analysis. With the rapid advancements in 3D imaging systems and the availability of excellent hardware and software support, 3D deep learning methods are gaining popularity in medical image analysis.
  arXiv  Detail & Related papers  (2021-08-19T03:23:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.