MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

• URL: http://arxiv.org/abs/2603.23067v2
• Date: Wed, 25 Mar 2026 07:41:44 GMT
• Title: MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding
• Abstract summary: Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context.<n>Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding.<n>We introduce textbfMLLMHWSI, a WSI-level MLLM that aligns visual features with multi-scale language at four distinct scales.
• Score: 39.29898673779214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce \textbf{MLLM-HWSI}, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight \textit{Cell-Cell Attention Fusion (CCAF)} transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: \href{https://github.com/BasitAlawode/HWSI-MLLM}{GitHub}.

Related papers

• Spatial Message Passing in Language Space for Pathology Image Interpretation [11.068039815749488]
  We introduce a framework that performs spatial reasoning entirely in language space, human-readable by construction.<n>SLMP represents a WSI region as a spatial text graph.<n>For each tile, an LLM refines its description by integrating language messages from adjacent tiles under a shared aggregation policy.<n>On representative HER2 and CAMELYON16 regions, SLMP improves tile-level tumor description accuracy in settings spanning general-purpose and pathology-specialized backbones.
  arXiv  Detail & Related papers  (2026-08-14T13:48:15Z)
• From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion [91.35078719566472]
  Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection.<n>We introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities.
  arXiv  Detail & Related papers  (2026-01-15T18:59:10Z)
• LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models [19.89635786218384]
  Whole Slide Image (WSI) understanding is fundamentally challenging due to its gigapixel scale and the extreme sparsity of diagnostically relevant regions.<n>Existing slide-level multimodal large language models (MLLMs) for pathology rely on heavy slide-level encoders.<n>We introduce an efficient MLLM framework, called LoC-Path, that replaces the expensive slide-level encoder with redundancy-reducing modules.
  arXiv  Detail & Related papers  (2025-12-05T03:16:46Z)
• Towards Fine-Grained Vision-Language Alignment for Few-Shot Anomaly Detection [65.29550320117526]
  We propose a novel framework named FineGrainedAD to improve anomaly localization performance.<n> Experiments demonstrate that the proposed FineGrainedAD achieves superior overall performance in few-shot settings.
  arXiv  Detail & Related papers  (2025-10-30T13:09:00Z)
• How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding [39.342366994703376]
  We introduce a probing framework to analyze how MLLMs process visual and textual inputs across layers.<n>We show that while the overall stage-wise structure remains stable across variations in visual tokenization, instruction tuning data, and pretraining corpus, the specific layer allocation to each stage shifts.
  arXiv  Detail & Related papers  (2025-08-27T21:22:01Z)
• LISA: A Layer-wise Integration and Suppression Approach for Hallucination Mitigation in Multimodal Large Language Models [8.122679857175315]
  Multimodal Large Language Models (MLLMs) excel in vision-language tasks but remain prone to object hallucinations.<n>We propose textbfLISA, which enhances generation consistency through hierarchical modulation and multi-layer fusion.<n>Experiments show that LISA reduces hallucinations by up to 53.6% in $mathrmCHAIR_I$ and improves POPE F1 by 4.5%.
  arXiv  Detail & Related papers  (2025-07-25T09:48:23Z)
• Multimodal Language Models See Better When They Look Shallower [54.5303326937134]
  Multimodal large language models (MLLMs) typically extract visual features from the final layers of a pretrained Vision Transformer (ViT)<n>We present the first comprehensive study of visual layer selection for MLLMs, analyzing representation similarity across ViT layers.<n>We find that while deep layers excel in semantic-rich tasks like OCR, shallow and middle layers significantly outperform them on fine-grained visual tasks.
  arXiv  Detail & Related papers  (2025-04-30T09:07:10Z)
• Slide-Level Prompt Learning with Vision Language Models for Few-Shot Multiple Instance Learning in Histopathology [21.81603581614496]
  We address the challenge of few-shot classification in histopathology whole slide images (WSIs)<n>Our method distinguishes itself by utilizing pathological prior knowledge from language models to identify crucial local tissue types (patches) for WSI classification.<n>Our approach effectively aligns patch images with tissue types, and we fine-tune our model via prompt learning using only a few labeled WSIs per category.
  arXiv  Detail & Related papers  (2025-03-21T15:40:37Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension [62.40482764691584]
  We introduce SEED-Bench-2-Plus, a benchmark specifically designed for evaluating textbftext-rich visual comprehension of MLLMs. Our benchmark comprises 2.3K multiple-choice questions with precise human annotations, spanning three broad categories: Charts, Maps, and Webs. We conduct a thorough evaluation involving 34 prominent MLLMs and emphasize the current limitations of MLLMs in text-rich visual comprehension.
  arXiv  Detail & Related papers  (2024-04-25T17:39:35Z)
• From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models [36.41816380074965]
  We investigate the effectiveness of different vision encoders within Large Language Models (MLLMs) Our findings reveal that the shallow layer features of CLIP offer particular advantages for fine-grained tasks such as grounding and region understanding. We propose a simple yet effective feature merging strategy, named COMM, that integrates CLIP and DINO with Multi-level features Merging.
  arXiv  Detail & Related papers  (2023-10-13T02:41:55Z)
• A Variational Hierarchical Model for Neural Cross-Lingual Summarization [85.44969140204026]
  Cross-lingual summarization () is to convert a document in one language to a summary in another one. Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model. We propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder.
  arXiv  Detail & Related papers  (2022-03-08T02:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.