Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities

• URL: http://arxiv.org/abs/2407.10785v1
• Date: Mon, 15 Jul 2024 15:03:01 GMT
• Title: Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities
• Authors: Nhat Le, Ciyue Shen, Chintan Shah, Blake Martin, Daniel Shenker, Harshith Padigela, Jennifer Hipp, Sean Grullon, John Abel, Harsha Vardhan Pokkalla, Dinkar Juyal,
• Abstract summary: We analyze the features from a ViT-Small encoder obtained from a pathology Foundation Model via application to two datasets. We discover an interpretable representation of cell and tissue morphology, along with gene expression within the model embedding space.
• Score: 1.4602325266401266
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Mechanistic interpretability has been explored in detail for large language models (LLMs). For the first time, we provide a preliminary investigation with similar interpretability methods for medical imaging. Specifically, we analyze the features from a ViT-Small encoder obtained from a pathology Foundation Model via application to two datasets: one dataset of pathology images, and one dataset of pathology images paired with spatial transcriptomics. We discover an interpretable representation of cell and tissue morphology, along with gene expression within the model embedding space. Our work paves the way for further exploration around interpretable feature dimensions and their utility for medical and clinical applications.

Related papers

• Multiplex Imaging Analysis in Pathology: a Comprehensive Review on Analytical Approaches and Digital Toolkits [0.7968706282619793]
  Multi multiplexed imaging allows for simultaneous visualization of multiple biomarkers in a single section. Data from multiplexed imaging requires sophisticated computational methods for preprocessing, segmentation, feature extraction, and spatial analysis. PathML is an AI-powered platform that streamlines image analysis, making complex interpretation accessible for clinical and research settings.
  arXiv  Detail & Related papers  (2024-11-01T18:02:41Z)
• Histopathology image embedding based on foundation models features aggregation for patient treatment response prediction [0.0]
  We propose a novel methodology for predicting Diffuse Large B-Cell Lymphoma patients treatment response from Whole Slide Images. Our method exploits several foundation models as feature extractors to obtain a local representation of the image corresponding to a small region of the tissue. Our experimental study conducted on a dataset of 152 patients, shows the promising results of our methodology.
  arXiv  Detail & Related papers  (2024-07-23T13:31:12Z)
• Anatomy-guided Pathology Segmentation [56.883822515800205]
  We develop a generalist segmentation model that combines anatomical and pathological information, aiming to enhance the segmentation accuracy of pathological features. Our Anatomy-Pathology Exchange (APEx) training utilizes a query-based segmentation transformer which decodes a joint feature space into query-representations for human anatomy. In doing so, we are able to report the best results across the board on FDG-PET-CT and Chest X-Ray pathology segmentation tasks with a margin of up to 3.3% as compared to strong baseline methods.
  arXiv  Detail & Related papers  (2024-07-08T11:44:15Z)
• PLUTO: Pathology-Universal Transformer [4.920983796208486]
  We propose PathoLogy Universal TransfOrmer (PLUTO): a light-weight pathology FM that is pre-trained on a diverse dataset of 195 million image tiles. We design task-specific adaptation heads that utilize PLUTO's output embeddings for tasks which span pathology scales. We find that PLUTO matches or outperforms existing task-specific baselines and pathology-specific foundation models.
  arXiv  Detail & Related papers  (2024-05-13T16:40:17Z)
• Knowledge-enhanced Visual-Language Pretraining for Computational Pathology [68.6831438330526]
  We consider the problem of visual representation learning for computational pathology, by exploiting large-scale image-text pairs gathered from public resources. We curate a pathology knowledge tree that consists of 50,470 informative attributes for 4,718 diseases requiring pathology diagnosis from 32 human tissues.
  arXiv  Detail & Related papers  (2024-04-15T17:11:25Z)
• HistoSegCap: Capsules for Weakly-Supervised Semantic Segmentation of Histological Tissue Type in Whole Slide Images [19.975420988169454]
  Digital pathology involves converting physical tissue slides into high-resolution Whole Slide Images (WSIs) Large histology slides with numerous microscopic fields pose challenges for visual search. Computer Aided Diagnosis (CAD) systems offer visual assistance in efficiently examining WSIs and identifying diagnostically relevant regions.
  arXiv  Detail & Related papers  (2024-02-16T17:44:11Z)
• HistoCartography: A Toolkit for Graph Analytics in Digital Pathology [0.6299766708197883]
  HistoCartography is a standardized python API with necessary preprocessing, machine learning and explainability tools to facilitate graph-analytics in computational pathology. We have benchmarked the computational time and performance on multiple datasets across different imaging types and histopathology tasks.
  arXiv  Detail & Related papers  (2021-07-21T13:34:14Z)
• MIMO: Mutual Integration of Patient Journey and Medical Ontology for Healthcare Representation Learning [49.57261599776167]
  We propose an end-to-end robust Transformer-based solution, Mutual Integration of patient journey and Medical Ontology (MIMO) for healthcare representation learning and predictive analytics.
  arXiv  Detail & Related papers  (2021-07-20T07:04:52Z)
• Data-driven generation of plausible tissue geometries for realistic photoacoustic image synthesis [53.65837038435433]
  Photoacoustic tomography (PAT) has the potential to recover morphological and functional tissue properties. We propose a novel approach to PAT data simulation, which we refer to as "learning to simulate" We leverage the concept of Generative Adversarial Networks (GANs) trained on semantically annotated medical imaging data to generate plausible tissue geometries.
  arXiv  Detail & Related papers  (2021-03-29T11:30:18Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.