Evidence-based diagnostic reasoning with multi-agent copilot for human pathology

• URL: http://arxiv.org/abs/2506.20964v1
• Date: Thu, 26 Jun 2025 03:02:16 GMT
• Title: Evidence-based diagnostic reasoning with multi-agent copilot for human pathology
• Authors: Chengkuan Chen, Luca L. Weishaupt, Drew F. K. Williamson, Richard J. Chen, Tong Ding, Bowen Chen, Anurag Vaidya, Long Phi Le, Guillaume Jaume, Ming Y. Lu, Faisal Mahmood,
• Abstract summary: Current multimodal large language models (MLLMs) in computational pathology face limitations.<n>We introduce PathChat+, a new MLLM specifically designed for human pathology, trained on over 1 million diverse, pathology-specific instruction samples.<n>We also present SlideSeek, a reasoning-enabled multi-agent AI system leveraging PathChat+ to autonomously evaluate gigapixel whole-slide images.
• Score: 7.976907866539546
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Pathology is experiencing rapid digital transformation driven by whole-slide imaging and artificial intelligence (AI). While deep learning-based computational pathology has achieved notable success, traditional models primarily focus on image analysis without integrating natural language instruction or rich, text-based context. Current multimodal large language models (MLLMs) in computational pathology face limitations, including insufficient training data, inadequate support and evaluation for multi-image understanding, and a lack of autonomous, diagnostic reasoning capabilities. To address these limitations, we introduce PathChat+, a new MLLM specifically designed for human pathology, trained on over 1 million diverse, pathology-specific instruction samples and nearly 5.5 million question answer turns. Extensive evaluations across diverse pathology benchmarks demonstrated that PathChat+ substantially outperforms the prior PathChat copilot, as well as both state-of-the-art (SOTA) general-purpose and other pathology-specific models. Furthermore, we present SlideSeek, a reasoning-enabled multi-agent AI system leveraging PathChat+ to autonomously evaluate gigapixel whole-slide images (WSIs) through iterative, hierarchical diagnostic reasoning, reaching high accuracy on DDxBench, a challenging open-ended differential diagnosis benchmark, while also capable of generating visually grounded, humanly-interpretable summary reports.

Related papers

• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• ForenX: Towards Explainable AI-Generated Image Detection with Multimodal Large Language Models [82.04858317800097]
  We present ForenX, a novel method that not only identifies the authenticity of images but also provides explanations that resonate with human thoughts.<n>ForenX employs the powerful multimodal large language models (MLLMs) to analyze and interpret forensic cues.<n>We introduce ForgReason, a dataset dedicated to descriptions of forgery evidences in AI-generated images.
  arXiv  Detail & Related papers  (2025-08-02T15:21:26Z)
• A Survey of Multimodal Ophthalmic Diagnostics: From Task-Specific Approaches to Foundational Models [28.34025112894094]
  The review focuses on two main categories: task-specific multimodal approaches and large-scale multimodal foundation models.<n>The survey critically examines important datasets, evaluation metrics, and methodological innovations.<n>It also discusses ongoing challenges such as variability in data, limited annotations, lack of interpretability, and issues with generalizability across different patient populations.
  arXiv  Detail & Related papers  (2025-07-31T10:49:21Z)
• Interpretable and Reliable Detection of AI-Generated Images via Grounded Reasoning in MLLMs [43.08776932101172]
  We build a dataset of AI-generated images annotated with bounding boxes and descriptive captions.<n>We then finetune MLLMs through a multi-stage optimization strategy.<n>The resulting model achieves superior performance in both detecting AI-generated images and localizing visual flaws.
  arXiv  Detail & Related papers  (2025-06-08T08:47:44Z)
• CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic [12.75486013022629]
  We introduce CPathAgent, an agent-based model that mimics pathologists' reasoning processes by autonomously executing zoom-in/out and navigation operations.<n>CPathAgent consistently outperforms existing approaches across three scales of benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T20:22:19Z)
• Advancing Conversational Diagnostic AI with Multimodal Reasoning [44.1996223689966]
  Articulate Medical Intelligence Explorer (AMIE)<n>System implements a state-aware dialogue framework, where conversation flow is dynamically controlled by intermediate model outputs.<n>We compared AMIE to primary care physicians (PCPs) in a randomized, blinded, OSCE-style study of chat-based consultations with patient actors.
  arXiv  Detail & Related papers  (2025-05-06T20:52:01Z)
• Exploring the Feasibility of Multimodal Chatbot AI as Copilot in Pathology Diagnostics: Generalist Model's Pitfall [17.9731336178034]
  ChatGPT and other multimodal models have shown promise in transforming medical image analysis through capabilities such as medical vision-language question answering. This study benchmarks the performance of GPT on pathology images, assessing their diagnostic accuracy and efficiency in real-word clinical records. We observe significant deficits of GPT in bone diseases and a fair-level performance in diseases from other three systems.
  arXiv  Detail & Related papers  (2024-09-04T01:30:05Z)
• Knowledge-enhanced Visual-Language Pretraining for Computational Pathology [68.6831438330526]
  We consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources. We curate a pathology knowledge tree that consists of 50,470 informative attributes for 4,718 diseases requiring pathology diagnosis from 32 human tissues.
  arXiv  Detail & Related papers  (2024-04-15T17:11:25Z)
• Segmentation, Classification and Interpretation of Breast Cancer Medical Images using Human-in-the-Loop Machine Learning [37.69303106863453]
  The paper deals with the integration of genomic data and Whole Slide Imaging (WSI) analysis of breast cancer. The involvement of a pathologist helped us to develop a better segmentation model and to enhance the explainatory capabilities of the models, but the classification results were suboptimal.
  arXiv  Detail & Related papers  (2024-03-29T10:49:02Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• A Foundational Multimodal Vision Language AI Assistant for Human Pathology [6.759775793033743]
  We present PathChat, a vision-language generalist AI assistant for human pathology using an in-house developed vision encoder pretrained on 100 million histology images. PathChat achieved a diagnostic accuracy of 87% on multiple-choice questions based on publicly available cases of diverse tissue origins and disease models.
  arXiv  Detail & Related papers  (2023-12-13T00:24:37Z)
• Towards a Visual-Language Foundation Model for Computational Pathology [5.72536252929528]
  We introduce CONtrastive learning from Captions for Histopathology (CONCH) CONCH is a visual-language foundation model developed using diverse sources of histopathology images, biomedical text, and task-agnostic pretraining. It is evaluated on a suite of 13 diverse benchmarks, achieving state-of-the-art performance on histology image classification, segmentation, captioning, text-to-image and image-to-text retrieval.
  arXiv  Detail & Related papers  (2023-07-24T16:13:43Z)
• PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology [15.419350834457136]
  We present PathAsst, a multimodal generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. The development of PathAsst involves three pivotal steps: data acquisition, CLIP model adaptation, and the training of PathAsst's multimodal generative capabilities. The experimental results of PathAsst show the potential of harnessing AI-powered generative foundation model to improve pathology diagnosis and treatment processes.
  arXiv  Detail & Related papers  (2023-05-24T11:55:50Z)
• Pixel-Level Explanation of Multiple Instance Learning Models in Biomedical Single Cell Images [52.527733226555206]
  We investigate the use of four attribution methods to explain a multiple instance learning models. We study two datasets of acute myeloid leukemia with over 100 000 single cell images. We compare attribution maps with the annotations of a medical expert to see how the model's decision-making differs from the human standard.
  arXiv  Detail & Related papers  (2023-03-15T14:00:11Z)
• OncoPetNet: A Deep Learning based AI system for mitotic figure counting on H&E stained whole slide digital images in a large veterinary diagnostic lab setting [47.38796928990688]
  Multiple state-of-the-art deep learning techniques for histopathology image classification and mitotic figure detection were used in the development of OncoPetNet. The proposed system, demonstrated significantly improved mitotic counting performance for 41 cancer cases across 14 cancer types compared to human expert baselines. In deployment, an effective 0.27 min/slide inference was achieved in a high throughput veterinary diagnostic service across 2 centers processing 3,323 digital whole slide images daily.
  arXiv  Detail & Related papers  (2021-08-17T20:01:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.