MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education

• URL: http://arxiv.org/abs/2605.04772v1
• Date: Wed, 06 May 2026 11:20:08 GMT
• Title: MIRAGE: Retrieval and Generation of Multimodal Images and Texts for Medical Education
• Abstract summary: MIRAGE is a multimodal medical text and image retrieval and generation system.<n>It allows users to find and generate clinically relevant images from trustworthy sources.<n>The system is accessible to medical students worldwide without requiring local computational resources or technical expertise.
• Score: 2.765006963244799
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Access to diverse, well-annotated medical images with interactive learning tools is fundamental for training practitioners in medicine and related fields to improve their diagnostic skills and understanding of anatomical structures. While medical atlases are valuable, they are often impractical due to their size and lack of interactivity, whereas online image search may provide mislabeled or incomplete material. To address this, we propose MIRAGE, a multimodal medical text and image retrieval and generation system that allows users to find and generate clinically relevant images from trustworthy sources by mapping both text and images to a shared latent space, enabling semantically meaningful queries. The system is based on a fine-tuned medical version of CLIP (MedICaT-ROCO), trained with the ROCO dataset, obtained from PubMed Central. MIRAGE allows users to give prompts to retrieve images, generate synthetic ones through a medical diffusion model (Prompt2MedImage) and receive enriched descriptions from a large language model (Dolly-v2-3b). It also supports a dual search option, enabling the visual comparison of different medical conditions. A key advantage of the system is that it relies entirely on publicly available pretrained models, ensuring reproducibility and accessibility. Our goal is to provide a free, transparent and easy-to-use didactic tool for medical students, especially those without programming skills. The system features an interface that enables interactive and personalized visual learning through medical image retrieval and generation. The system is accessible to medical students worldwide without requiring local computational resources or technical expertise, and is currently deployed on Kaggle: http://www-vpu.eps.uam.es/mirage

Related papers

• UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis [41.864457631668806]
  We introduce UniMedVL, the first medical unified multimodal model for the simultaneous analysis of image understanding and generation tasks.<n>UniMedVL achieves superior performance on five medical image understanding benchmarks, while matching specialized models in generation quality across eight medical imaging modalities.
  arXiv  Detail & Related papers  (2025-10-17T14:54:58Z)
• RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining [64.66825253356869]
  We propose a novel methodology that leverages dense radiology reports to define image-wise similarity ordering at multiple granularities.<n>We construct two comprehensive medical imaging retrieval datasets: MIMIC-IR for Chest X-rays and CTRATE-IR for CT scans.<n>We develop two retrieval systems, RadIR-CXR and model-ChestCT, which demonstrate superior performance in traditional image-image and image-report retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-06T17:43:03Z)
• A Survey of Medical Vision-and-Language Applications and Their Techniques [48.268198631277315]
  Medical vision-and-language models (MVLMs) have attracted substantial interest due to their capability to offer a natural language interface for interpreting complex medical data. Here, we provide a comprehensive overview of MVLMs and the various medical tasks to which they have been applied. We also examine the datasets used for these tasks and compare the performance of different models based on standardized evaluation metrics.
  arXiv  Detail & Related papers  (2024-11-19T03:27:05Z)
• Autoregressive Sequence Modeling for 3D Medical Image Representation [48.706230961589924]
  We introduce a pioneering method for learning 3D medical image representations through an autoregressive sequence pre-training framework.<n>Our approach various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence.
  arXiv  Detail & Related papers  (2024-09-13T10:19:10Z)
• MedDiT: A Knowledge-Controlled Diffusion Transformer Framework for Dynamic Medical Image Generation in Virtual Simulated Patient [38.2964748653908]
  This paper introduces MedDiT, a novel knowledge-controlled conversational framework that can generate plausible medical images aligned with simulated patient symptoms. MedDiT integrates various patient Knowledge Graphs (KGs), which describe the attributes and symptoms of patients, to dynamically prompt Large Language Models' (LLMs) behavior. A well-tuned Diffusion Transformer (DiT) model is incorporated to generate medical images according to the specified patient attributes in the KG.
  arXiv  Detail & Related papers  (2024-08-22T09:10:29Z)
• AliFuse: Aligning and Fusing Multi-modal Medical Data for Computer-Aided Diagnosis [1.64647940449869]
  We propose a transformer-based framework, called Alifuse, for aligning and fusing multimodal medical data.<n>We convert medical images and both unstructured and structured clinical records into vision and language tokens.<n>We apply Alifuse to classify Alzheimer's disease, achieving state-of-the-art performance on five public datasets and outperforming eight baselines.
  arXiv  Detail & Related papers  (2024-01-02T07:28:21Z)
• Unified Medical Image Pre-training in Language-Guided Common Semantic Space [39.61770813855078]
  We propose an Unified Medical Image Pre-training framework, namely UniMedI. UniMedI uses diagnostic reports as common semantic space to create unified representations for diverse modalities of medical images. We evaluate its performance on both 2D and 3D images across 10 different datasets.
  arXiv  Detail & Related papers  (2023-11-24T22:01:12Z)
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day [85.19963303642427]
  We propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics. This enables us to train a Large Language and Vision Assistant for BioMedicine in less than 15 hours (with eight A100s)
  arXiv  Detail & Related papers  (2023-06-01T16:50:07Z)
• Sketch-based Medical Image Retrieval [37.009806114204636]
  We introduce a novel sketch-based medical image retrieval system that enables users to find images of interest without example images. Key idea lies in feature decomposition of medical images, whereby the entire feature of a medical image can be decomposed into and reconstructed from normal and abnormal features. Our system achieves flexible medical image retrieval on demand, thereby expanding the utility of medical image databases.
  arXiv  Detail & Related papers  (2023-03-07T03:41:13Z)
• Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge [68.90835997085557]
  We propose a systematic and effective approach to enhance structured medical knowledge from three perspectives. First, we align the representations of the vision encoder and the language encoder through knowledge. Second, we inject knowledge into the multi-modal fusion model to enable the model to perform reasoning using knowledge as the supplementation of the input image and text. Third, we guide the model to put emphasis on the most critical information in images and texts by designing knowledge-induced pretext tasks.
  arXiv  Detail & Related papers  (2022-09-15T08:00:01Z)
• Cross-Modal Information Maximization for Medical Imaging: CMIM [62.28852442561818]
  In hospitals, data are siloed to specific information systems that make the same information available under different modalities. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time. We propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time.
  arXiv  Detail & Related papers  (2020-10-20T20:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.