A Data and Compute Efficient Design for Limited-Resources Deep Learning

• URL: http://arxiv.org/abs/2004.09691v2
• Date: Wed, 8 Jul 2020 11:29:18 GMT
• Title: A Data and Compute Efficient Design for Limited-Resources Deep Learning
• Authors: Mirgahney Mohamed, Gabriele Cesa, Taco S. Cohen and Max Welling
• Abstract summary: equivariant neural networks have gained increased interest in the deep learning community. They have been successfully applied in the medical domain where symmetries in the data can be effectively exploited to build more accurate and robust models. Mobile, on-device implementations of deep learning solutions have been developed for medical applications. However, equivariant models are commonly implemented using large and computationally expensive architectures, not suitable to run on mobile devices. In this work, we design and test an equivariant version of MobileNetV2 and further optimize it with model quantization to enable more efficient inference.
• Score: 68.55415606184
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Thanks to their improved data efficiency, equivariant neural networks have gained increased interest in the deep learning community. They have been successfully applied in the medical domain where symmetries in the data can be effectively exploited to build more accurate and robust models. To be able to reach a much larger body of patients, mobile, on-device implementations of deep learning solutions have been developed for medical applications. However, equivariant models are commonly implemented using large and computationally expensive architectures, not suitable to run on mobile devices. In this work, we design and test an equivariant version of MobileNetV2 and further optimize it with model quantization to enable more efficient inference. We achieve close-to state of the art performance on the Patch Camelyon (PCam) medical dataset while being more computationally efficient.

Related papers

• Optimizing Dense Feed-Forward Neural Networks [0.0]
  We propose a novel feed-forward neural network constructing method based on pruning and transfer learning. Our approach can compress the number of parameters by more than 70%. We also evaluate the transfer learning level comparing the refined model and the original one training from scratch a neural network.
  arXiv  Detail & Related papers  (2023-12-16T23:23:16Z)
• E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation [36.367368163120794]
  We propose a 3D medical image segmentation model, named Efficient to Efficient Network (E2ENet) It incorporates two parametrically and computationally efficient designs. It consistently achieves a superior trade-off between accuracy and efficiency across various resource constraints.
  arXiv  Detail & Related papers  (2023-12-07T22:13:37Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines [49.75878234192369]
  We present WEAVER, a simple, yet efficient post-processing method that infuses old knowledge into the new model. We show that applying WEAVER in a sequential manner results in similar word embedding distributions as doing a combined training on all data at once.
  arXiv  Detail & Related papers  (2022-02-21T10:34:41Z)
• Conservative Objective Models for Effective Offline Model-Based Optimization [78.19085445065845]
  Computational design problems arise in a number of settings, from synthetic biology to computer architectures. We propose a method that learns a model of the objective function that lower bounds the actual value of the ground-truth objective on out-of-distribution inputs. COMs are simple to implement and outperform a number of existing methods on a wide range of MBO problems.
  arXiv  Detail & Related papers  (2021-07-14T17:55:28Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• FastSal: a Computationally Efficient Network for Visual Saliency Prediction [7.742198347952173]
  We show that MobileNetV2 makes an excellent backbone for a visual saliency model and can be effective even without a complex decoder. We also show that knowledge transfer from a more computationally expensive model like DeepGaze II can be achieved via pseudo-labelling an unlabelled dataset.
  arXiv  Detail & Related papers  (2020-08-25T16:32:33Z)
• Real-time Human Activity Recognition Using Conditionally Parametrized Convolutions on Mobile and Wearable Devices [14.260179062012512]
  deep convolutional neural networks (CNNs) have achieved state-of-the-art performance on various HAR datasets. A high number of operations in deep leaning increases computational cost and is not suitable for real-time HAR using mobile and wearable sensors. We propose a efficient CNN using conditionally parametrized convolution for real-time HAR on mobile and wearable devices.
  arXiv  Detail & Related papers  (2020-06-05T07:06:42Z)
• Efficient Training of Deep Convolutional Neural Networks by Augmentation in Embedding Space [24.847651341371684]
  In applications where data are scarce, transfer learning and data augmentation techniques are commonly used to improve the generalization of deep learning models. Fine-tuning a transfer model with data augmentation in the raw input space has a high computational cost to run the full network for every augmented input. We propose a method that replaces the augmentation in the raw input space with an approximate one that acts purely in the embedding space.
  arXiv  Detail & Related papers  (2020-02-12T03:26:33Z)
• An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devices [58.62801151916888]
  We introduce a new sparsity dimension, namely pattern-based sparsity that comprises pattern and connectivity sparsity, and becoming both highly accurate and hardware friendly. Our approach on the new pattern-based sparsity naturally fits into compiler optimization for highly efficient DNN execution on mobile platforms.
  arXiv  Detail & Related papers  (2020-01-20T16:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.