More From Less: Self-Supervised Knowledge Distillation for Routine Histopathology Data

• URL: http://arxiv.org/abs/2303.10656v2
• Date: Fri, 21 Jul 2023 17:15:24 GMT
• Title: More From Less: Self-Supervised Knowledge Distillation for Routine Histopathology Data
• Authors: Lucas Farndale, Robert Insall and Ke Yuan
• Abstract summary: We show that it is possible to distil knowledge during training from information-dense data into models which only require information-sparse data for inference. This improves downstream classification accuracy on information-sparse data, making it comparable with the fully-supervised baseline. This approach enables the design of models which require only routine images, but contain insights from state-of-the-art data, allowing better use of the available resources.
• Score: 3.93181912653522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical imaging technologies are generating increasingly large amounts of high-quality, information-dense data. Despite the progress, practical use of advanced imaging technologies for research and diagnosis remains limited by cost and availability, so information-sparse data such as H&E stains are relied on in practice. The study of diseased tissue requires methods which can leverage these information-dense data to extract more value from routine, information-sparse data. Using self-supervised deep learning, we demonstrate that it is possible to distil knowledge during training from information-dense data into models which only require information-sparse data for inference. This improves downstream classification accuracy on information-sparse data, making it comparable with the fully-supervised baseline. We find substantial effects on the learned representations, and this training process identifies subtle features which otherwise go undetected. This approach enables the design of models which require only routine images, but contain insights from state-of-the-art data, allowing better use of the available resources.

Related papers

• Local-to-Global Self-Supervised Representation Learning for Diabetic Retinopathy Grading [0.0]
  This research aims to present a novel hybrid learning model using self-supervised learning and knowledge distillation. In our algorithm, for the first time among all self-supervised learning and knowledge distillation models, the test dataset is 50% larger than the training dataset. Compared to a similar state-of-the-art model, our results achieved higher accuracy and more effective representation spaces.
  arXiv  Detail & Related papers  (2024-10-01T15:19:16Z)
• Extracting Training Data from Unconditional Diffusion Models [76.85077961718875]
  diffusion probabilistic models (DPMs) are being employed as mainstream models for generative artificial intelligence (AI) We aim to establish a theoretical understanding of memorization in DPMs with 1) a memorization metric for theoretical analysis, 2) an analysis of conditional memorization with informative and random labels, and 3) two better evaluation metrics for measuring memorization. Based on the theoretical analysis, we propose a novel data extraction method called textbfSurrogate condItional Data Extraction (SIDE) that leverages a trained on generated data as a surrogate condition to extract training data directly from unconditional diffusion models.
  arXiv  Detail & Related papers  (2024-06-18T16:20:12Z)
• Less is more: Ensemble Learning for Retinal Disease Recognition Under Limited Resources [12.119196313470887]
  This paper introduces a novel ensemble learning mechanism designed for recognizing retinal diseases under limited resources. The mechanism leverages insights from multiple pre-trained models, facilitating the transfer and adaptation of their knowledge to Retinal OCT images.
  arXiv  Detail & Related papers  (2024-02-15T06:58:25Z)
• Rapid Training Data Creation by Synthesizing Medical Images for Classification and Localization [10.506584969668792]
  We present a method for the transformation of real data to train any Deep Neural Network to solve the above problems. For the weakly supervised model, we show that the localization accuracy increases significantly using the generated data. In the latter model, we show that the accuracy, when trained with generated images, closely parallels the accuracy when trained with exhaustively annotated real images.
  arXiv  Detail & Related papers  (2023-08-09T03:49:12Z)
• Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images [2.5075774184834803]
  Deep neural networks still rely on large amounts of training data to avoid overfitting. Labeled training data for real-world applications such as healthcare is limited and difficult to access. We build upon the emerging success of text-to-image diffusion probabilistic models in augmenting the training samples of our macroscopic skin disease dataset.
  arXiv  Detail & Related papers  (2023-01-12T04:22:23Z)
• Knowledge-Guided Data-Centric AI in Healthcare: Progress, Shortcomings, and Future Directions [7.673853485227739]
  Deep learning is able to learn a wide range of examples of a particular concept or meaning. In medicine, having a diverse set of training data on a particular disease can lead to the development of a model that is able to accurately predict the disease. Despite the potential benefits, there have not been significant advances in image-based diagnosis due to a lack of high-quality annotated data. This article highlights the importance of using a data-centric approach to improve the quality of data representations.
  arXiv  Detail & Related papers  (2022-12-27T19:35:14Z)
• BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines [49.75878234192369]
  We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once.
  arXiv  Detail & Related papers  (2022-02-21T10:34:41Z)
• Human-in-the-Loop Disinformation Detection: Stance, Sentiment, or Something Else? [93.91375268580806]
  Both politics and pandemics have recently provided ample motivation for the development of machine learning-enabled disinformation (a.k.a. fake news) detection algorithms. Existing literature has focused primarily on the fully-automated case, but the resulting techniques cannot reliably detect disinformation on the varied topics, sources, and time scales required for military applications. By leveraging an already-available analyst as a human-in-the-loop, canonical machine learning techniques of sentiment analysis, aspect-based sentiment analysis, and stance detection become plausible methods to use for a partially-automated disinformation detection system.
  arXiv  Detail & Related papers  (2021-11-09T13:30:34Z)
• Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation [51.66271681532262]
  Online Self-Acquired Knowledge Distillation (OSAKD) is proposed, aiming to improve the performance of any deep neural model in an online manner. We utilize k-nn non-parametric density estimation technique for estimating the unknown probability distributions of the data samples in the output feature space.
  arXiv  Detail & Related papers  (2021-08-26T14:01:04Z)
• Data Consistent CT Reconstruction from Insufficient Data with Learned Prior Images [70.13735569016752]
  We investigate the robustness of deep learning in CT image reconstruction by showing false negative and false positive lesion cases. We propose a data consistent reconstruction (DCR) method to improve their image quality, which combines the advantages of compressed sensing and deep learning. The efficacy of the proposed method is demonstrated in cone-beam CT with truncated data, limited-angle data and sparse-view data, respectively.
  arXiv  Detail & Related papers  (2020-05-20T13:30:49Z)
• Self-Training with Improved Regularization for Sample-Efficient Chest X-Ray Classification [80.00316465793702]
  We present a deep learning framework that enables robust modeling in challenging scenarios. Our results show that using 85% lesser labeled data, we can build predictive models that match the performance of classifiers trained in a large-scale data setting.
  arXiv  Detail & Related papers  (2020-05-03T02:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.