SCoTTi: Save Computation at Training Time with an adaptive framework

• URL: http://arxiv.org/abs/2312.12483v1
• Date: Tue, 19 Dec 2023 16:19:33 GMT
• Title: SCoTTi: Save Computation at Training Time with an adaptive framework
• Authors: Ziyu Lin, Enzo Tartaglione, Van-Tam Nguyen
• Abstract summary: On-device training is an emerging approach in machine learning where models are trained on edge devices. We propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the challenge of reducing resource consumption during training. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks.
• Score: 7.780766187171572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.

Related papers

• Neural Horizon Model Predictive Control -- Increasing Computational Efficiency with Neural Networks [0.0]
  We propose a proposed machine-learning supported approach to model predictive control. We propose approximating part of the problem horizon, while maintaining safety guarantees. The proposed MPC scheme can be applied to a wide range of applications, including those requiring a rapid control response.
  arXiv  Detail & Related papers  (2024-08-19T08:13:37Z)
• Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning [88.78080749909665]
  Current on-device training methods just focus on efficient training without considering the catastrophic forgetting. This paper proposes a simple but effective edge-friendly incremental learning framework. Our method achieves average accuracy boost of 38.08% with even less memory and approximate computation.
  arXiv  Detail & Related papers  (2024-06-13T05:49:29Z)
• Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks [9.96381061452642]
  We propose Sparse Spectral Training (SST), an advanced training methodology that updates all singular values and selectively updates singular vectors of network weights. SST refines the training process by employing a targeted updating strategy for singular vectors, which is determined by a multinomial sampling method weighted by the significance of the singular values. On OPT-125M, with rank equating to 8.3% of embedding dimension, SST reduces the perplexity gap to full-rank training by 67.6%, demonstrating a significant reduction of the performance loss with prevalent low-rank methods.
  arXiv  Detail & Related papers  (2024-05-24T11:59:41Z)
• Always-Sparse Training by Growing Connections with Guided Stochastic Exploration [46.4179239171213]
  We propose an efficient always-sparse training algorithm with excellent scaling to larger and sparser models. We evaluate our method on CIFAR-10/100 and ImageNet using VGG, and ViT models, and compare it against a range of sparsification methods.
  arXiv  Detail & Related papers  (2024-01-12T21:32:04Z)
• Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening [51.34904967046097]
  Selective Synaptic Dampening (SSD) is a fast, performant, and does not require long-term storage of the training data. We present a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data.
  arXiv  Detail & Related papers  (2023-08-15T11:30:45Z)
• Dynamic Scheduling for Federated Edge Learning with Streaming Data [56.91063444859008]
  We consider a Federated Edge Learning (FEEL) system where training data are randomly generated over time at a set of distributed edge devices with long-term energy constraints. Due to limited communication resources and latency requirements, only a subset of devices is scheduled for participating in the local training process in every iteration.
  arXiv  Detail & Related papers  (2023-05-02T07:41:16Z)
• Towards Compute-Optimal Transfer Learning [82.88829463290041]
  We argue that zero-shot structured pruning of pretrained models allows them to increase compute efficiency with minimal reduction in performance. Our results show that pruning convolutional filters of pretrained models can lead to more than 20% performance improvement in low computational regimes.
  arXiv  Detail & Related papers  (2023-04-25T21:49:09Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Paging [35.397804171588476]
  Fine-tuning models on edge devices would enable privacy-preserving personalization over sensitive data. We present POET, an algorithm to enable training large neural networks on memory-scarce battery-operated edge devices.
  arXiv  Detail & Related papers  (2022-07-15T18:36:29Z)
• Low-Rank Training of Deep Neural Networks for Emerging Memory Technology [4.456122555367167]
  We address two key challenges for training on edge devices with non-volatile memory: low write density and low auxiliary memory. We present a low-rank training scheme that addresses these challenges while maintaining computational efficiency.
  arXiv  Detail & Related papers  (2020-09-08T17:59:56Z)
• Predicting Training Time Without Training [120.92623395389255]
  We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function. We leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model. We are able to predict the time it takes to fine-tune a model to a given loss without having to perform any training.
  arXiv  Detail & Related papers  (2020-08-28T04:29:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.