Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities

• URL: http://arxiv.org/abs/2407.10785v1
• Date: Mon, 15 Jul 2024 15:03:01 GMT
• Title: Interpretability analysis on a pathology foundation model reveals biologically relevant embeddings across modalities
• Authors: Nhat Le, Ciyue Shen, Chintan Shah, Blake Martin, Daniel Shenker, Harshith Padigela, Jennifer Hipp, Sean Grullon, John Abel, Harsha Vardhan Pokkalla, Dinkar Juyal,
• Abstract summary: We analyze the features from a ViT-Small encoder obtained from a pathology Foundation Model via application to two datasets. We discover an interpretable representation of cell and tissue morphology, along with gene expression within the model embedding space.
• Score: 1.4602325266401266
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Mechanistic interpretability has been explored in detail for large language models (LLMs). For the first time, we provide a preliminary investigation with similar interpretability methods for medical imaging. Specifically, we analyze the features from a ViT-Small encoder obtained from a pathology Foundation Model via application to two datasets: one dataset of pathology images, and one dataset of pathology images paired with spatial transcriptomics. We discover an interpretable representation of cell and tissue morphology, along with gene expression within the model embedding space. Our work paves the way for further exploration around interpretable feature dimensions and their utility for medical and clinical applications.

Related papers

• Causal Representation Learning from Multimodal Biological Observations [57.00712157758845]
  We aim to develop flexible identification conditions for multimodal data. We establish identifiability guarantees for each latent component, extending the subspace identification results from prior work. Our key theoretical ingredient is the structural sparsity of the causal connections among distinct modalities.
  arXiv  Detail & Related papers  (2024-11-10T16:40:27Z)
• Deep Latent Variable Modeling of Physiological Signals [0.8702432681310401]
  We explore high-dimensional problems related to physiological monitoring using latent variable models. First, we present a novel deep state-space model to generate electrical waveforms of the heart using optically obtained signals as inputs. Second, we present a brain signal modeling scheme that combines the strengths of probabilistic graphical models and deep adversarial learning. Third, we propose a framework for the joint modeling of physiological measures and behavior.
  arXiv  Detail & Related papers  (2024-05-29T17:07:33Z)
• Tertiary Lymphoid Structures Generation through Graph-based Diffusion [54.37503714313661]
  In this work, we leverage state-of-the-art graph-based diffusion models to generate biologically meaningful cell-graphs. We show that the adopted graph diffusion model is able to accurately learn the distribution of cells in terms of their tertiary lymphoid structures (TLS) content.
  arXiv  Detail & Related papers  (2023-10-10T14:37:17Z)
• Biologically-informed deep learning models for cancer: fundamental trends for encoding and interpreting oncology data [0.0]
  We provide a structured literature analysis focused on Deep Learning (DL) models used to support inference in cancer biology. The work focuses on how existing models address the need for better dialogue with prior knowledge, biological plausibility and interpretability.
  arXiv  Detail & Related papers  (2022-07-02T12:11:35Z)
• Learning multi-scale functional representations of proteins from single-cell microscopy data [77.34726150561087]
  We show that simple convolutional networks trained on localization classification can learn protein representations that encapsulate diverse functional information. We also propose a robust evaluation strategy to assess quality of protein representations across different scales of biological function.
  arXiv  Detail & Related papers  (2022-05-24T00:00:07Z)
• PathologyBERT -- Pre-trained Vs. A New Transformer Language Model for Pathology Domain [2.3628956573813498]
  Successful text mining of a large pathology database can play a critical role to advance 'big data' cancer research. No pathology-specific language space exist to support the rapid data-mining development in pathology space. PathologyBERT is a pre-trained masked language model which was trained on 347,173 histopathology specimen reports.
  arXiv  Detail & Related papers  (2022-05-13T20:42:07Z)
• Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unlabeled, unannotated pathology slides [9.27127895781971]
  Histomorphological Phenotype Learning operates via the automatic discovery of discriminatory image features in small image tiles. Tiles are grouped into morphologically similar clusters which constitute a library of histomorphological phenotypes.
  arXiv  Detail & Related papers  (2022-05-04T08:06:55Z)
• Analyzing the Effects of Handling Data Imbalance on Learned Features from Medical Images by Looking Into the Models [50.537859423741644]
  Training a model on an imbalanced dataset can introduce unique challenges to the learning problem. We look deeper into the internal units of neural networks to observe how handling data imbalance affects the learned features.
  arXiv  Detail & Related papers  (2022-04-04T09:38:38Z)
• Self-Supervised Vision Transformers Learn Visual Concepts in Histopathology [5.164102666113966]
  We conduct a search for good representations in pathology by training a variety of self-supervised models with validation on a variety of weakly-supervised and patch-level tasks. Our key finding is in discovering that Vision Transformers using DINO-based knowledge distillation are able to learn data-efficient and interpretable features in histology images.
  arXiv  Detail & Related papers  (2022-03-01T16:14:41Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.