HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

• URL: http://arxiv.org/abs/2603.19957v1
• Date: Fri, 20 Mar 2026 13:58:02 GMT
• Title: HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction
• Abstract summary: HiPath is a lightweight pathology vision-language framework built on frozen UNI2 and Qwen3 backbones.<n>Trained on 749K real-world Chinese pathology cases, HiPath achieves 68.9% strict and 74.7% clinically acceptable accuracy with a 97.3% safety rate.
• Score: 5.151955122293638
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pathology reports are structured, multi-granular documents encoding diagnostic conclusions, histological grades, and ancillary test results across one or more anatomical sites; yet existing pathology vision-language models (VLMs) reduce this output to a flat label or free-form text. We present HiPath, a lightweight VLM framework built on frozen UNI2 and Qwen3 backbones that treats structured report prediction as its primary training objective. Three trainable modules totalling 15M parameters address complementary aspects of the problem: a Hierarchical Patch Aggregator (HiPA) for multi-image visual encoding, Hierarchical Contrastive Learning (HiCL) for cross-modal alignment via optimal transport, and Slot-based Masked Diagnosis Prediction (Slot-MDP) for structured diagnosis generation. Trained on 749K real-world Chinese pathology cases from three hospitals, HiPath achieves 68.9% strict and 74.7% clinically acceptable accuracy with a 97.3% safety rate, outperforming all baselines under the same frozen backbone. Cross-hospital evaluation confirms generalisation with only a 3.4pp drop in strict accuracy while maintaining 97.1% safety.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs [0.0]
  We evaluate three open-source models, Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B, via local deployment across 320 clinical scenarios spanning ten medical specialties.<n> schema noncompliance is consistent across the three model families, with baseline compliance rates ranging from 85.9 to 91.6 percent.<n> validation-repair framework achieves 99.0 percent overall compliance, ranging from 98.4 to 99.4 percent across models.
  arXiv  Detail & Related papers  (2026-07-27T12:45:27Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models [3.6659638651482536]
  Recent advances in 3D medical vision-language models have enabled joint reasoning over volumetric images and text.<n>It remains unclear whether these models learn spatially grounded anatomy from 3D volumes or rely primarily on learned priors and language correlations.<n>We introduce CT-SpatialVQA, a benchmark designed to evaluate semantic-spatial reasoning in 3D CT data.
  arXiv  Detail & Related papers  (2026-05-09T08:16:00Z)
• CTIS-QA: Clinical Template-Informed Slide-level Question Answering for Pathology [14.647193383163284]
  We introduce a clinical diagnosis template-based pipeline to systematically collect and structure pathological information.<n>CTIS-QA is a Slide-level Question Answering model, featuring a dual-stream architecture that mimics pathologists' diagnostic approach.
  arXiv  Detail & Related papers  (2026-01-05T03:54:02Z)
• Beyond Pixel Simulation: Pathology Image Generation via Diagnostic Semantic Tokens and Prototype Control [45.749134892112714]
  We introduce UniPath, a semantics-driven pathology image generation framework.<n>UniPath implements Multi-Stream Control: a Raw-Text stream; a High-Level Semantics stream that uses learnable queries to a frozen pathology MLLM.<n>On the data front, we curate a 2.65M image-text corpus and a finely annotated, high-quality 68K subset to alleviate data scarcity.<n>Experiments demonstrate UniPath's SOTA performance, including a Patho-FID of 80.9 (51% better than the second-best) and fine-grained semantic control achieving 98.7% of the real-
  arXiv  Detail & Related papers  (2025-12-24T08:52:08Z)
• MedForget: Hierarchy-Aware Multimodal Unlearning Testbed for Medical AI [66.0701326117134]
  MedForget is a hierarchy-aware multimodal unlearning testbed for building compliant medical AI systems.<n>We show that existing methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance.<n>We introduce a reconstruction attack that progressively adds hierarchical level context to prompts.
  arXiv  Detail & Related papers  (2025-12-10T17:55:06Z)
• MedXplain-VQA: Multi-Component Explainable Medical Visual Question Answering [1.4413073343064953]
  We present MedXplain-VQA, a comprehensive framework integrating five explainable AI components to deliver interpretable medical image analysis.<n>The framework leverages a fine-tuned BLIP-2 backbone, medical query reformulation, enhanced Grad-CAM attention, precise region extraction, and structured chain-of-thought reasoning via multi-modal language models.
  arXiv  Detail & Related papers  (2025-10-26T19:23:20Z)
• YpathRAG:A Retrieval-Augmented Generation Framework and Benchmark for Pathology [16.03995342015096]
  We build a pathology vector database covering 28 subfields and 1.53 million paragraphs.<n>We present YpathRAG, a pathology-oriented RAG framework with dual-channel hybrid retrieval.<n>We also release two evaluation benchmarks, YpathR and YpathQA-M.
  arXiv  Detail & Related papers  (2025-10-07T08:47:59Z)
• The Next Layer: Augmenting Foundation Models with Structure-Preserving and Attention-Guided Learning for Local Patches to Global Context Awareness in Computational Pathology [23.32822092398391]
  We present EAGLE-Net, a structure-preserving, attention-guided MIL architecture designed to augment prediction and interpretability.<n>We benchmarked it on large pan-cancer datasets, including 3 cancer types for classification (10,260 slides) and 7 cancer types for survival prediction (4,172 slides)
  arXiv  Detail & Related papers  (2025-08-27T14:19:38Z)
• HySemRAG: A Hybrid Semantic Retrieval-Augmented Generation Framework for Automated Literature Synthesis and Methodological Gap Analysis [55.2480439325792]
  HySemRAG is a framework that combines Extract, Transform, Load (ETL) pipelines with Retrieval-Augmented Generation (RAG)<n>System addresses limitations in existing RAG architectures through a multi-layered approach.
  arXiv  Detail & Related papers  (2025-08-01T20:30:42Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• SemioLLM: Evaluating Large Language Models for Diagnostic Reasoning from Unstructured Clinical Narratives in Epilepsy [45.2233252981348]
  Large Language Models (LLMs) have been shown to encode clinical knowledge.<n>We present SemioLLM, an evaluation framework that benchmarks 6 state-of-the-art models.<n>We show that most LLMs are able to accurately and confidently generate probabilistic predictions of seizure onset zones in the brain.
  arXiv  Detail & Related papers  (2024-07-03T11:02:12Z)
• Self-supervised Answer Retrieval on Clinical Notes [68.87777592015402]
  We introduce CAPR, a rule-based self-supervision objective for training Transformer language models for domain-specific passage matching. We apply our objective in four Transformer-based architectures: Contextual Document Vectors, Bi-, Poly- and Cross-encoders. We report that CAPR outperforms strong baselines in the retrieval of domain-specific passages and effectively generalizes across rule-based and human-labeled passages.
  arXiv  Detail & Related papers  (2021-08-02T10:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.