Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs

• URL: http://arxiv.org/abs/2604.25296v1
• Date: Tue, 28 Apr 2026 07:05:26 GMT
• Title: Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs
• Abstract summary: Multimodal Large Language Models (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies.<n>We propose a novel Entity-Centric Medical Data Engineering framework.<n>We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository.
• Score: 34.87373678921793
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies that rely on coarse-grained partitioning by modality or department. Such fragmented approaches fail to capture the hierarchical and interconnected nature of clinical medical knowledge, limiting the models' ability to perform fine-grained recognition and complex reasoning. In this paper, we propose a novel Entity-Centric Medical Data Engineering framework. We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository. Building upon the MET, we propose an advanced data engine that includes: (1) node-guided retrieval to anchor raw data to specific medical concepts, (2) a two-stage hybrid filtering and alignment pipeline to ensure precise visual-semantic correspondence, and (3) knowledge-aware data synthesis to generate enriched captions and targeted reasoning VQA pairs, leveraging structural constraints. Extensive evaluations across six medical benchmarks demonstrate that our approach significantly enhances the medical capabilities of general-purpose MLLMs, improving their ability to handle complex clinical queries and achieve state-of-the-art performance in diverse medical contexts.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding [67.57715989349471]
  Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge.<n>We introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding.<n>We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks.
  arXiv  Detail & Related papers  (2026-07-27T17:59:49Z)
• MED-COPILOT: A Medical Assistant Powered by GraphRAG and Similar Patient Case Retrieval [12.265116154395434]
  We present MED-COPILOT, an interactive clinical decision-support system designed for clinicians and medical trainees.<n>The system builds a structured knowledge graph from WHO and NICE guidelines, applies community-level summarization for efficient retrieval, and maintains a 36,000-case similar-patient database.
  arXiv  Detail & Related papers  (2026-02-28T04:32:03Z)
• MedVerse: Efficient and Reliable Medical Reasoning via DAG-Structured Parallel Execution [63.128360383691295]
  We propose MedVerse, a reasoning framework for complex medical inference.<n>For data creation, we introduce the MedVerse Curator, which synthesizes knowledge-grounded medical reasoning paths.<n>We develop a customized inference engine that supports parallel execution without additional overhead.
  arXiv  Detail & Related papers  (2026-02-07T12:54:01Z)
• Evaluating Strategies for Synthesizing Clinical Notes for Medical Multimodal AI [1.8648093673053043]
  Multimodal (MM) learning is emerging as a promising paradigm in biomedical artificial intelligence (AI) applications.<n>This work investigates strategies for generating synthetic textual clinical notes, in terms of prompt design and medical metadata inclusion.
  arXiv  Detail & Related papers  (2025-11-26T19:01:12Z)
• Utilizing Large Language Models for Zero-Shot Medical Ontology Extension from Clinical Notes [13.564947974902429]
  We propose CLOZE, a novel framework that uses large language models (LLMs) to automatically extract medical entities from clinical notes.<n>By capitalizing on the strong language understanding and extensive knowledge of pre-trained LLMs, CLOZE effectively identifies disease-related concepts and captures complex hierarchical relationships.
  arXiv  Detail & Related papers  (2025-11-20T17:00:46Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning [57.873833577058]
  We build a multimodal dataset enriched with extensive medical knowledge.<n>We then introduce our medical-specialized MLLM: Lingshu.<n>Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities.
  arXiv  Detail & Related papers  (2025-06-08T08:47:30Z)
• MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Book [5.318470975871017]
  We present MedBookVQA, a systematic and comprehensive multimodal benchmark derived from open-access medical textbooks.<n>We generate 5,000 clinically relevant questions spanning modality recognition, classification, anatomical identification, symptom diagnosis, and surgical procedures.<n>We evaluate a wide array of MLLMs, including proprietary, open-sourced, medical, and reasoning models, revealing significant performance disparities across task types and model categories.
  arXiv  Detail & Related papers  (2025-06-01T06:28:36Z)
• Large Language Models for Biomedical Knowledge Graph Construction: Information extraction from EMR notes [0.0]
  We propose an end-to-end machine learning solution based on large language models (LLMs) The entities used in the KG construction process are diseases, factors, treatments, as well as manifestations that coexist with the patient while experiencing the disease. The application of the proposed methodology is demonstrated on age-related macular degeneration.
  arXiv  Detail & Related papers  (2023-01-29T15:52:33Z)
• Self-supervised Answer Retrieval on Clinical Notes [68.87777592015402]
  We introduce CAPR, a rule-based self-supervision objective for training Transformer language models for domain-specific passage matching. We apply our objective in four Transformer-based architectures: Contextual Document Vectors, Bi-, Poly- and Cross-encoders. We report that CAPR outperforms strong baselines in the retrieval of domain-specific passages and effectively generalizes across rule-based and human-labeled passages.
  arXiv  Detail & Related papers  (2021-08-02T10:42:52Z)
• A Meta-embedding-based Ensemble Approach for ICD Coding Prediction [64.42386426730695]
  International Classification of Diseases (ICD) are the de facto codes used globally for clinical coding. These codes enable healthcare providers to claim reimbursement and facilitate efficient storage and retrieval of diagnostic information. Our proposed approach enhances the performance of neural models by effectively training word vectors using routine medical data as well as external knowledge from scientific articles.
  arXiv  Detail & Related papers  (2021-02-26T17:49:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.