Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervision

• URL: http://arxiv.org/abs/2308.10064v1
• Date: Sat, 19 Aug 2023 15:57:19 GMT
• Title: Efficient Representation Learning for Healthcare with Cross-Architectural Self-Supervision
• Authors: Pranav Singh and Jacopo Cirrone
• Abstract summary: We present Cross Architectural - Self Supervision (CASS) in response to this challenge. We show that CASS-trained CNNs and Transformers outperform existing self-supervised learning methods across four diverse healthcare datasets. We also demonstrate that CASS is considerably more robust to variations in batch size and pretraining epochs, making it a suitable candidate for machine learning in healthcare applications.
• Score: 5.439020425819001
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In healthcare and biomedical applications, extreme computational requirements pose a significant barrier to adopting representation learning. Representation learning can enhance the performance of deep learning architectures by learning useful priors from limited medical data. However, state-of-the-art self-supervised techniques suffer from reduced performance when using smaller batch sizes or shorter pretraining epochs, which are more practical in clinical settings. We present Cross Architectural - Self Supervision (CASS) in response to this challenge. This novel siamese self-supervised learning approach synergistically leverages Transformer and Convolutional Neural Networks (CNN) for efficient learning. Our empirical evaluation demonstrates that CASS-trained CNNs and Transformers outperform existing self-supervised learning methods across four diverse healthcare datasets. With only 1% labeled data for finetuning, CASS achieves a 3.8% average improvement; with 10% labeled data, it gains 5.9%; and with 100% labeled data, it reaches a remarkable 10.13% enhancement. Notably, CASS reduces pretraining time by 69% compared to state-of-the-art methods, making it more amenable to clinical implementation. We also demonstrate that CASS is considerably more robust to variations in batch size and pretraining epochs, making it a suitable candidate for machine learning in healthcare applications.

Related papers

• An Efficient Contrastive Unimodal Pretraining Method for EHR Time Series Data [35.943089444017666]
  We propose an efficient method of contrastive pretraining tailored for long clinical timeseries data. Our model demonstrates the ability to impute missing measurements, providing clinicians with deeper insights into patient conditions.
  arXiv  Detail & Related papers  (2024-10-11T19:05:25Z)
• Transfer learning from a sparsely annotated dataset of 3D medical images [4.477071833136902]
  This study explores the use of transfer learning to improve the performance of deep convolutional neural networks for organ segmentation in medical imaging. A base segmentation model was trained on a large and sparsely annotated dataset; its weights were used for transfer learning on four new down-stream segmentation tasks. The results showed that transfer learning from the base model was beneficial when small datasets were available.
  arXiv  Detail & Related papers  (2023-11-08T21:31:02Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Cross-Architectural Positive Pairs improve the effectiveness of Self-Supervised Learning [0.0]
  Cross Architectural - Self Supervision (CASS) is a novel self-supervised learning approach that leverages Transformer and CNN simultaneously. We show that CASS-trained CNNs and Transformers across four diverse datasets gained an average of 3.8% with 1% labeled data. We also show that CASS is much more robust to changes in batch size and training epochs than existing state-of-the-art self-supervised learning approaches.
  arXiv  Detail & Related papers  (2023-01-27T23:27:24Z)
• Dissecting Self-Supervised Learning Methods for Surgical Computer Vision [51.370873913181605]
  Self-Supervised Learning (SSL) methods have begun to gain traction in the general computer vision community. The effectiveness of SSL methods in more complex and impactful domains, such as medicine and surgery, remains limited and unexplored. We present an extensive analysis of the performance of these methods on the Cholec80 dataset for two fundamental and popular tasks in surgical context understanding, phase recognition and tool presence detection.
  arXiv  Detail & Related papers  (2022-07-01T14:17:11Z)
• CASS: Cross Architectural Self-Supervision for Medical Image Analysis [0.0]
  Cross Architectural Self-Supervision is a novel self-supervised learning approach which leverages transformers and CNN simultaneously. Compared to existing state-of-the-art self-supervised learning approaches, we empirically show CASS trained CNNs, and Transformers gained an average of 8.5% with 100% labelled data.
  arXiv  Detail & Related papers  (2022-06-08T21:25:15Z)
• Robust and Efficient Medical Imaging with Self-Supervision [80.62711706785834]
  We present REMEDIS, a unified representation learning strategy to improve robustness and data-efficiency of medical imaging AI. We study a diverse range of medical imaging tasks and simulate three realistic application scenarios using retrospective data.
  arXiv  Detail & Related papers  (2022-05-19T17:34:18Z)
• BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines [49.75878234192369]
  We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once.
  arXiv  Detail & Related papers  (2022-02-21T10:34:41Z)
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy [70.71457102672545]
  We compare the impact of different training procedures for diabetic retinopathy grading. We investigate different aspects such as quantitative performance, statistics of the learned feature representations, interpretability and robustness to image distortions. Our results indicate that models from ImageNet pretraining report a significant increase in performance, generalization and robustness to image distortions.
  arXiv  Detail & Related papers  (2021-06-25T08:32:45Z)
• A Machine Learning Early Warning System: Multicenter Validation in Brazilian Hospitals [4.659599449441919]
  Early recognition of clinical deterioration is one of the main steps for reducing inpatient morbidity and mortality. Since hospital wards are given less attention compared to the Intensive Care Unit, ICU, we hypothesized that when a platform is connected to a stream of EHR, there would be a drastic improvement in dangerous situations awareness. With the application of machine learning, the system is capable to consider all patient's history and through the use of high-performing predictive models, an intelligent early warning system is enabled.
  arXiv  Detail & Related papers  (2020-06-09T21:21:38Z)
• Self-Training with Improved Regularization for Sample-Efficient Chest X-Ray Classification [80.00316465793702]
  We present a deep learning framework that enables robust modeling in challenging scenarios. Our results show that using 85% lesser labeled data, we can build predictive models that match the performance of classifiers trained in a large-scale data setting.
  arXiv  Detail & Related papers  (2020-05-03T02:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.