Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning

• URL: http://arxiv.org/abs/2403.14616v1
• Date: Thu, 21 Mar 2024 17:58:56 GMT
• Title: Hierarchical Text-to-Vision Self Supervised Alignment for Improved Histopathology Representation Learning
• Authors: Hasindri Watawana, Kanchana Ranasinghe, Tariq Mahmood, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan,
• Abstract summary: We present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets.
• Score: 64.1316997189396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised representation learning has been highly promising for histopathology image analysis with numerous approaches leveraging their patient-slide-patch hierarchy to learn better representations. In this paper, we explore how the combination of domain specific natural language information with such hierarchical visual representations can benefit rich representation learning for medical image tasks. Building on automated language description generation for features visible in histopathology images, we present a novel language-tied self-supervised learning framework, Hierarchical Language-tied Self-Supervision (HLSS) for histopathology images. We explore contrastive objectives and granular language description based text alignment at multiple hierarchies to inject language modality information into the visual representations. Our resulting model achieves state-of-the-art performance on two medical imaging benchmarks, OpenSRH and TCGA datasets. Our framework also provides better interpretability with our language aligned representation space. Code is available at https://github.com/Hasindri/HLSS.

Related papers

• Compositional Entailment Learning for Hyperbolic Vision-Language Models [54.41927525264365]
  We show how to fully leverage the innate hierarchical nature of hyperbolic embeddings by looking beyond individual image-text pairs. We propose Compositional Entailment Learning for hyperbolic vision-language models. Empirical evaluation on a hyperbolic vision-language model trained with millions of image-text pairs shows that the proposed compositional learning approach outperforms conventional Euclidean CLIP learning.
  arXiv  Detail & Related papers  (2024-10-09T14:12:50Z)
• Emergent Visual-Semantic Hierarchies in Image-Text Representations [13.300199242824934]
  We study the knowledge of existing foundation models, finding that they exhibit emergent understanding of visual-semantic hierarchies. We propose the Radial Embedding (RE) framework for probing and optimizing hierarchical understanding.
  arXiv  Detail & Related papers  (2024-07-11T14:09:42Z)
• CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment [40.811510317145675]
  CPLIP is a new unsupervised technique to enhance the alignment of images and text in histopathology. evaluated across multiple histopathology tasks, CPLIP shows notable improvements in zero-shot learning scenarios. To encourage further research and replication, the code for CPLIP is available on GitHub.
  arXiv  Detail & Related papers  (2024-06-07T18:39:58Z)
• XCoOp: Explainable Prompt Learning for Computer-Aided Diagnosis via Concept-guided Context Optimization [4.634780391920529]
  We propose a novel explainable prompt learning framework that leverages medical knowledge by aligning the semantics of images, learnable prompts, and clinical concept-driven prompts. Our framework addresses the lack of valuable concept annotations by eliciting knowledge from large language models. Our method simultaneously achieves superior diagnostic performance, flexibility, and interpretability, shedding light on the effectiveness of foundation models in facilitating XAI.
  arXiv  Detail & Related papers  (2024-03-14T14:02:01Z)
• Anatomical Structure-Guided Medical Vision-Language Pre-training [21.68719061251635]
  We propose an Anatomical Structure-Guided (ASG) framework for learning medical visual representations. For anatomical region, we design an automatic anatomical region-sentence alignment paradigm in collaboration with radiologists. For finding and existence, we regard them as image tags, applying an image-tag recognition decoder to associate image features with their respective tags within each sample.
  arXiv  Detail & Related papers  (2024-03-14T11:29:47Z)
• Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models [24.456117679941816]
  Contrastive Reading Model (Cream) is a novel neural architecture designed to enhance the language-image understanding capability of Large Language Models (LLMs) Our approach bridges the gap between vision and language understanding, paving the way for the development of more sophisticated Document Intelligence Assistants.
  arXiv  Detail & Related papers  (2023-05-24T11:59:13Z)
• Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality [50.48859793121308]
  Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. Recent research has highlighted severe limitations in their ability to perform compositional reasoning over objects, attributes, and relations.
  arXiv  Detail & Related papers  (2023-05-23T08:28:38Z)
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision [57.031588264841]
  We leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme.
  arXiv  Detail & Related papers  (2021-02-11T10:08:12Z)
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language [148.0843278195794]
  We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
  arXiv  Detail & Related papers  (2020-11-18T20:21:19Z)
• Improving Image Captioning with Better Use of Captions [65.39641077768488]
  We present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences.
  arXiv  Detail & Related papers  (2020-06-21T14:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.