TensAIR: Real-Time Training of Neural Networks from Data-streams

• URL: http://arxiv.org/abs/2211.10280v2
• Date: Thu, 18 Apr 2024 15:07:35 GMT
• Title: TensAIR: Real-Time Training of Neural Networks from Data-streams
• Authors: Mauro D. L. Tosi, Vinu E. Venugopal, Martin Theobald,
• Abstract summary: This paper presents TensAIR, the first OL system for training ANNs in real time. TensAIR achieves remarkable performance and scalability by using a decentralized and asynchronous architecture to train ANN models. We empirically demonstrate that TensAIR achieves a nearly linear scale-out performance in terms of (1) the number of worker nodes deployed in the network, and (2) the throughput at which the data batches arrive.
• Score: 1.409180142531996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online learning (OL) from data streams is an emerging area of research that encompasses numerous challenges from stream processing, machine learning, and networking. Stream-processing platforms, such as Apache Kafka and Flink, have basic extensions for the training of Artificial Neural Networks (ANNs) in a stream-processing pipeline. However, these extensions were not designed to train ANNs in real-time, and they suffer from performance and scalability issues when doing so. This paper presents TensAIR, the first OL system for training ANNs in real time. TensAIR achieves remarkable performance and scalability by using a decentralized and asynchronous architecture to train ANN models (either freshly initialized or pre-trained) via DASGD (decentralized and asynchronous stochastic gradient descent). We empirically demonstrate that TensAIR achieves a nearly linear scale-out performance in terms of (1) the number of worker nodes deployed in the network, and (2) the throughput at which the data batches arrive at the dataflow operators. We depict the versatility of TensAIR by investigating both sparse (word embedding) and dense (image classification) use cases, for which TensAIR achieved from 6 to 116 times higher sustainable throughput rates than state-of-the-art systems for training ANN in a stream-processing pipeline.

Related papers

• Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness [15.841764541165238]
  Spiking Neural Networks (SNNs) allow for energy-efficient implementation on neuromorphic hardware. Currently, SNNs trained with current direct training approaches are constrained to a specific time step. We introduce Mixed Time-step Training (MTT), a novel method that improves the temporal flexibility of SNNs.
  arXiv  Detail & Related papers  (2025-03-18T06:09:42Z)
• Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learners [82.72552644267724]
  BoostPFN can outperform standard PFNs with the same size of training samples in large datasets. High performance is maintained for up to 50x of the pre-training size of PFNs.
  arXiv  Detail & Related papers  (2025-03-03T07:31:40Z)
• Estimating Post-Synaptic Effects for Online Training of Feed-Forward SNNs [0.27016900604393124]
  Facilitating online learning in spiking neural networks (SNNs) is a key step in developing event-based models. We propose Online Training with Postsynaptic Estimates (OTPE) for training feed-forward SNNs. We show improved scaling for multi-layer networks using a novel approximation of temporal effects on the subsequent layer's activity.
  arXiv  Detail & Related papers  (2023-11-07T16:53:39Z)
• Transferability of Convolutional Neural Networks in Stationary Learning Tasks [96.00428692404354]
  We introduce a novel framework for efficient training of convolutional neural networks (CNNs) for large-scale spatial problems. We show that a CNN trained on small windows of such signals achieves a nearly performance on much larger windows without retraining. Our results show that the CNN is able to tackle problems with many hundreds of agents after being trained with fewer than ten.
  arXiv  Detail & Related papers  (2023-07-21T13:51:45Z)
• Large Deviations for Accelerating Neural Networks Training [5.864710987890994]
  We propose the LAD Improved Iterative Training (LIIT), a novel training approach for ANN using large deviations principle. The LIIT approach uses a Modified Training Sample (MTS) that is generated and iteratively updated using a LAD anomaly score based sampling strategy. The MTS sample is designed to be well representative of the training data by including most anomalous of the observations in each class.
  arXiv  Detail & Related papers  (2023-03-02T04:14:05Z)
• Online Evolutionary Neural Architecture Search for Multivariate Non-Stationary Time Series Forecasting [72.89994745876086]
  This work presents the Online Neuro-Evolution-based Neural Architecture Search (ONE-NAS) algorithm. ONE-NAS is a novel neural architecture search method capable of automatically designing and dynamically training recurrent neural networks (RNNs) for online forecasting tasks. Results demonstrate that ONE-NAS outperforms traditional statistical time series forecasting methods.
  arXiv  Detail & Related papers  (2023-02-20T22:25:47Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Training Spiking Neural Networks with Local Tandem Learning [96.32026780517097]
  Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient than their predecessors. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL) We demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity.
  arXiv  Detail & Related papers  (2022-10-10T10:05:00Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better [88.28293442298015]
  Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. We develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network.
  arXiv  Detail & Related papers  (2021-12-18T02:26:38Z)
• Going Deeper With Directly-Trained Larger Spiking Neural Networks [20.40894876501739]
  Spiking neural networks (SNNs) are promising in coding for bio-usible information and event-driven signal processing. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. We propose a CIF-dependent batch normalization (tpladBN) method based on the emerging-temporal backproation threshold.
  arXiv  Detail & Related papers  (2020-10-29T07:15:52Z)
• On the performance of deep learning models for time series classification in streaming [0.0]
  This work is to assess the performance of different types of deep architectures for data streaming classification. We evaluate models such as multi-layer perceptrons, recurrent, convolutional and temporal convolutional neural networks over several time-series datasets.
  arXiv  Detail & Related papers  (2020-03-05T11:41:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.