ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images

• URL: http://arxiv.org/abs/2311.15264v2
• Date: Sat, 1 Jun 2024 16:28:16 GMT
• Title: ChAda-ViT : Channel Adaptive Attention for Joint Representation Learning of Heterogeneous Microscopy Images
• Authors: Nicolas Bourriez, Ihab Bendidi, Ethan Cohen, Gabriel Watkinson, Maxime Sanchez, Guillaume Bollot, Auguste Genovesio,
• Abstract summary: We propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture. We also introduce IDRCell100k, a bioimage dataset with a rich set of 79 experiments covering 7 microscope modalities. Our architecture, trained in a self-supervised manner, outperforms existing approaches in several biologically relevant downstream tasks.
• Score: 2.954116522244175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unlike color photography images, which are consistently encoded into RGB channels, biological images encompass various modalities, where the type of microscopy and the meaning of each channel varies with each experiment. Importantly, the number of channels can range from one to a dozen and their correlation is often comparatively much lower than RGB, as each of them brings specific information content. This aspect is largely overlooked by methods designed out of the bioimage field, and current solutions mostly focus on intra-channel spatial attention, often ignoring the relationship between channels, yet crucial in most biological applications. Importantly, the variable channel type and count prevent the projection of several experiments to a unified representation for large scale pre-training. In this study, we propose ChAda-ViT, a novel Channel Adaptive Vision Transformer architecture employing an Inter-Channel Attention mechanism on images with an arbitrary number, order and type of channels. We also introduce IDRCell100k, a bioimage dataset with a rich set of 79 experiments covering 7 microscope modalities, with a multitude of channel types, and counts varying from 1 to 10 per experiment. Our architecture, trained in a self-supervised manner, outperforms existing approaches in several biologically relevant downstream tasks. Additionally, it can be used to bridge the gap for the first time between assays with different microscopes, channel numbers or types by embedding various image and experimental modalities into a unified biological image representation. The latter should facilitate interdisciplinary studies and pave the way for better adoption of deep learning in biological image-based analyses. Code and Data available at https://github.com/nicoboou/chadavit.

Related papers

• Interpretable Embeddings for Segmentation-Free Single-Cell Analysis in Multiplex Imaging [1.8687965482996822]
  Multiplex Imaging (MI) enables the simultaneous visualization of multiple biological markers in separate imaging channels at subcellular resolution. We propose a segmentation-free deep learning approach that leverages grouped convolutions to learn interpretable embedded features from each imaging channel.
  arXiv  Detail & Related papers  (2024-11-02T11:21:33Z)
• Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers [18.731717752379232]
  Multi-Channel Imaging (MCI) models must support a variety of channel configurations at test time. Recent work has extended traditional visual encoders for MCI, such as Vision Transformers (ViT), by supplementing pixel information with an encoding representing the channel configuration. We propose DiChaViT, which aims to enhance the diversity in the learned features of MCI-ViT models.
  arXiv  Detail & Related papers  (2024-05-26T03:41:40Z)
• CHAMMI: A benchmark for channel-adaptive models in microscopy imaging [18.220276947512843]
  We present a benchmark for investigating channel-adaptive models in microscopy imaging. We find that channel-adaptive models can generalize better to out-of-domain tasks and can be computationally efficient.
  arXiv  Detail & Related papers  (2023-10-30T02:03:28Z)
• Affine-Consistent Transformer for Multi-Class Cell Nuclei Detection [76.11864242047074]
  We propose a novel Affine-Consistent Transformer (AC-Former), which directly yields a sequence of nucleus positions. We introduce an Adaptive Affine Transformer (AAT) module, which can automatically learn the key spatial transformations to warp original images for local network training. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art algorithms on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-22T02:27:02Z)
• Learning Channel Importance for High Content Imaging with Interpretable Deep Input Channel Mixing [1.2963000794351183]
  We present a novel approach which utilizes multi-spectral information of high content images to interpret a certain aspect of cellular biology. We introduce DCmix, a lightweight, scaleable and end-to-end trainable mixing layer which enables interpretable predictions in high content imaging.
  arXiv  Detail & Related papers  (2023-08-31T11:11:38Z)
• Unpaired Image-to-Image Translation with Limited Data to Reveal Subtle Phenotypes [0.5076419064097732]
  We present an improved CycleGAN architecture that employs self-supervised discriminators to alleviate the need for numerous images. We also provide results obtained with small biological datasets on obvious and non-obvious cell phenotype variations.
  arXiv  Detail & Related papers  (2023-01-21T16:25:04Z)
• Stain-invariant self supervised learning for histopathology image analysis [74.98663573628743]
  We present a self-supervised algorithm for several classification tasks within hematoxylin and eosin stained images of breast cancer. Our method achieves the state-of-the-art performance on several publicly available breast cancer datasets.
  arXiv  Detail & Related papers  (2022-11-14T18:16:36Z)
• Affinity Feature Strengthening for Accurate, Complete and Robust Vessel Segmentation [48.638327652506284]
  Vessel segmentation is crucial in many medical image applications, such as detecting coronary stenoses, retinal vessel diseases and brain aneurysms. We present a novel approach, the affinity feature strengthening network (AFN), which jointly models geometry and refines pixel-wise segmentation features using a contrast-insensitive, multiscale affinity approach.
  arXiv  Detail & Related papers  (2022-11-12T05:39:17Z)
• Two-Stream Graph Convolutional Network for Intra-oral Scanner Image Segmentation [133.02190910009384]
  We propose a two-stream graph convolutional network (i.e., TSGCN) to handle inter-view confusion between different raw attributes. Our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation.
  arXiv  Detail & Related papers  (2022-04-19T10:41:09Z)
• Interpreting Class Conditional GANs with Channel Awareness [57.01413866290279]
  We investigate how a class conditional generator unifies the synthesis of multiple classes. To describe such a phenomenon, we propose channel awareness, which quantitatively characterizes how a single channel contributes to the final synthesis. Our algorithm enables several novel applications with conditional GANs.
  arXiv  Detail & Related papers  (2022-03-21T17:53:22Z)
• Channel Interaction Networks for Fine-Grained Image Categorization [61.095320862647476]
  Fine-grained image categorization is challenging due to the subtle inter-class differences. We propose a channel interaction network (CIN), which models the channel-wise interplay both within an image and across images. Our model can be trained efficiently in an end-to-end fashion without the need of multi-stage training and testing.
  arXiv  Detail & Related papers  (2020-03-11T11:51:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.