PePR: Performance Per Resource Unit as a Metric to Promote Small-Scale Deep Learning in Medical Image Analysis

• URL: http://arxiv.org/abs/2403.12562v2
• Date: Thu, 05 Dec 2024 11:57:19 GMT
• Title: PePR: Performance Per Resource Unit as a Metric to Promote Small-Scale Deep Learning in Medical Image Analysis
• Authors: Raghavendra Selvan, Bob Pepin, Christian Igel, Gabrielle Samuel, Erik B Dam,
• Abstract summary: Large-scale resources have been used to train progressively larger deep learning models. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such scale. We take a comprehensive look at the landscape of existing DL models for medical image analysis tasks and demonstrate their usefulness in settings where resources are limited.
• Score: 6.474195280156721
• License:
• Abstract: The recent advances in deep learning (DL) have been accelerated by access to large-scale data and compute. These large-scale resources have been used to train progressively larger models which are resource intensive in terms of compute, data, energy, and carbon emissions. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such scale, particularly in the Global South. In this work, we take a comprehensive look at the landscape of existing DL models for medical image analysis tasks and demonstrate their usefulness in settings where resources are limited. To account for the resource consumption of DL models, we introduce a novel measure to estimate the performance per resource unit, which we call the PePR score. Using a diverse family of 131 unique DL architectures (spanning 1M to 130M trainable parameters) and three medical image datasets, we capture trends about the performance-resource trade-offs. In applications like medical image analysis, we argue that small-scale, specialized models are better than striving for large-scale models. Furthermore, we show that using existing pretrained models that are fine-tuned on new data can significantly reduce the computational resources and data required compared to training models from scratch. We hope this work will encourage the community to focus on improving AI equity by developing methods and models with smaller resource footprints.

Related papers

• MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies [85.57899012821211]
  Small Language Models (SLMs) are a resource-efficient alternative to Large Language Models (LLMs) We introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants. We also introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K.
  arXiv  Detail & Related papers  (2024-04-09T15:36:50Z)
• Adaptive Affinity-Based Generalization For MRI Imaging Segmentation Across Resource-Limited Settings [1.5703963908242198]
  This paper introduces a novel relation-based knowledge framework by seamlessly combining adaptive affinity-based and kernel-based distillation. To validate our innovative approach, we conducted experiments on publicly available multi-source prostate MRI data.
  arXiv  Detail & Related papers  (2024-04-03T13:35:51Z)
• Promising and worth-to-try future directions for advancing state-of-the-art surrogates methods of agent-based models in social and health computational sciences [0.0]
  Execution and runtime performance of model-based analysis tools for realistic large-scale ABMs can be excessively long. The main aim of this ad-hoc brief report is to highlight some of surrogate models that were adequate and computationally less demanding for nonlinear dynamical models.
  arXiv  Detail & Related papers  (2024-03-07T11:30:56Z)
• Less is more: Ensemble Learning for Retinal Disease Recognition Under Limited Resources [12.119196313470887]
  This paper introduces a novel ensemble learning mechanism designed for recognizing retinal diseases under limited resources. The mechanism leverages insights from multiple pre-trained models, facilitating the transfer and adaptation of their knowledge to Retinal OCT images.
  arXiv  Detail & Related papers  (2024-02-15T06:58:25Z)
• A Simple and Efficient Baseline for Data Attribution on Images [107.12337511216228]
  Current state-of-the-art approaches require a large ensemble of as many as 300,000 models to accurately attribute model predictions. In this work, we focus on a minimalist baseline, utilizing the feature space of a backbone pretrained via self-supervised learning to perform data attribution. Our method is model-agnostic and scales easily to large datasets.
  arXiv  Detail & Related papers  (2023-11-03T17:29:46Z)
• Reusing Pretrained Models by Multi-linear Operators for Efficient Training [65.64075958382034]
  Training large models from scratch usually costs a substantial amount of resources. Recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model. We propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model.
  arXiv  Detail & Related papers  (2023-10-16T06:16:47Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Delving Deeper into Data Scaling in Masked Image Modeling [145.36501330782357]
  We conduct an empirical study on the scaling capability of masked image modeling (MIM) methods for visual recognition. Specifically, we utilize the web-collected Coyo-700M dataset. Our goal is to investigate how the performance changes on downstream tasks when scaling with different sizes of data and models.
  arXiv  Detail & Related papers  (2023-05-24T15:33:46Z)
• Operating critical machine learning models in resource constrained regimes [0.18416014644193066]
  We investigate the trade-off between resource consumption and performance of deep learning models. Deep learning models are used in critical settings such as in clinics.
  arXiv  Detail & Related papers  (2023-03-17T12:02:08Z)
• Top-KAST: Top-K Always Sparse Training [50.05611544535801]
  We propose Top-KAST, a method that preserves constant sparsity throughout training. We show that it performs comparably to or better than previous works when training models on the established ImageNet benchmark. In addition to our ImageNet results, we also demonstrate our approach in the domain of language modeling.
  arXiv  Detail & Related papers  (2021-06-07T11:13:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.