DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation

• URL: http://arxiv.org/abs/2604.17209v1
• Date: Sun, 19 Apr 2026 02:40:57 GMT
• Title: DREAM: Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion for Expert Precision Medical Report Generation
• Abstract summary: DREAM is a novel framework for high-fidelity medical report generation.<n>It employs a unique two-stage fusion mechanism that intelligently integrates visual data with clinical keywords curated by ophthalmologists.<n>It sets a new state-of-the-art on the DeepEyeNet benchmark, achieving a BLEU-4 score of 0.241.
• Score: 3.1912286225194877
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automating medical reports for retinal images requires a sophisticated blend of visual pattern recognition and deep clinical knowledge. Current Large Vision-Language Models (LVLMs) often struggle in specialized medical fields where data is scarce, leading to models that overfit and miss subtle but critical pathologies. To address this, we introduce DREAM (Dynamic Retinal Enhancement with Adaptive Multi-modal Fusion), a novel framework for high-fidelity medical report generation that excels even with limited data. DREAM employs a unique two-stage fusion mechanism that intelligently integrates visual data with clinical keywords curated by ophthalmologists. First, the Abstractor module maps image and keyword features into a shared space, enhancing visual data with pathology-relevant insights. Next, the Adaptor performs adaptive multi-modal fusion, dynamically weighting the importance of each modality using learnable parameters to create a unified representation. To ensure the model's outputs are semantically grounded in clinical reality, a Contrastive Alignment module aligns these fused representations with ground-truth medical reports during training. By combining medical expertise with an efficient fusion strategy, DREAM sets a new state-of-the-art on the DeepEyeNet benchmark, achieving a BLEU-4 score of 0.241, and further demonstrates strong generalization to the ROCO dataset.

Related papers

• ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding [67.57715989349471]
  Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge.<n>We introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding.<n>We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks.
  arXiv  Detail & Related papers  (2026-07-27T17:59:49Z)
• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development [314.80153557710616]
  We present the largest survey to date of medical image datasets, covering over 1,000 open-access datasets.<n>Our analysis exposes a landscape that is modest in scale, fragmented across narrowly scoped tasks, and unevenly distributed across organs and modalities.<n>We propose a metadata-driven fusion paradigm (MDFP) that integrates public datasets with shared modalities or tasks.
  arXiv  Detail & Related papers  (2026-03-29T00:46:53Z)
• MedGround: Bridging the Evidence Gap in Medical Vision-Language Models with Verified Grounding Data [32.65971100171597]
  We introduce MedGround, an automated pipeline that transforms segmentation resources into high-quality medical referring grounding data.<n>We also present MedGround-35K, a novel multimodal medical dataset.
  arXiv  Detail & Related papers  (2026-01-11T10:34:18Z)
• Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models [45.285970665585914]
  We propose a comprehensive framework for Continual Learning.<n>We employ a multi-modal, multi-layer RAG system that provides real-time guidance for model fine-tuning.<n>We introduce a dynamic knowledge distillation framework.
  arXiv  Detail & Related papers  (2025-12-15T08:09:40Z)
• Multimodal Large Language Models for Medical Report Generation via Customized Prompt Tuning [20.195025131749944]
  We present MRG-LLM, a novel large language model (MLLM) that combines a frozen LLM with a learnable visual encoder.<n>We propose two implementations: prompt-wise and promptbook-wise customization, enabling precise and targeted report generation.
  arXiv  Detail & Related papers  (2025-06-18T14:09:34Z)
• PRETI: Patient-Aware Retinal Foundation Model via Metadata-Guided Representation Learning [3.771396977579353]
  PRETI is a retinal foundation model that integrates metadata-aware learning with robust self-supervised representation learning.<n>We construct patient-level data pairs, associating images from the same individual to improve robustness against non-clinical variations.<n>Experiments demonstrate PRETI achieves state-of-the-art results across diverse diseases and biomarker predictions.
  arXiv  Detail & Related papers  (2025-05-18T04:59:03Z)
• Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation [26.589728923739596]
  We introduce a framework specifically designed for multimodal medical data generation.<n>By enabling the generation of multi-view chest X-rays and their associated clinical report, it bridges the gap between general-purpose vision-language models and the specialized requirements of healthcare.<n>Our framework achieves comparable or even superior performance compared to real data on downstream disease classification tasks.
  arXiv  Detail & Related papers  (2025-05-02T08:07:24Z)
• ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models [95.47808515575382]
  ExGra-Med is a novel framework for vision-language integration in medical AI.<n>It aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence.<n>It matches LLaVA-Med's performance using just 10% of the pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance.
  arXiv  Detail & Related papers  (2024-10-03T15:52:03Z)
• ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features [54.37042005469384]
  We announce MVKL, the first multimodal mammography dataset encompassing multi-view images, detailed manifestations and reports. Based on this dataset, we focus on the challanging task of unsupervised pretraining. We propose ViKL, a framework that synergizes Visual, Knowledge, and Linguistic features.
  arXiv  Detail & Related papers  (2024-09-24T05:01:23Z)
• Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning [65.54680361074882]
  Eye-gaze Guided Multi-modal Alignment (EGMA) framework harnesses eye-gaze data for better alignment of medical visual and textual features. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets.
  arXiv  Detail & Related papers  (2024-03-19T03:59:14Z)
• LLM-driven Multimodal Target Volume Contouring in Radiation Oncology [46.23891509553877]
  Large language models (LLMs) can facilitate the integration of the textural information and images. We present a novel LLM-driven multimodal AI, namely LLMSeg, that is applicable to the challenging task of target volume contouring for radiation therapy. We demonstrate that the proposed model exhibits markedly improved performance compared to conventional unimodal AI models.
  arXiv  Detail & Related papers  (2023-11-03T13:38:42Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.