An introduction to distributed training of deep neural networks for
segmentation tasks with large seismic datasets
- URL: http://arxiv.org/abs/2102.13003v1
- Date: Thu, 25 Feb 2021 17:06:00 GMT
- Title: An introduction to distributed training of deep neural networks for
segmentation tasks with large seismic datasets
- Authors: Claire Birnie, Haithem Jarraya and Fredrik Hansteen
- Abstract summary: This paper illustrates how to tackle the two main issues of training of large neural networks: memory limitations and impracticably large training times.
We show how over 750GB of data can be used to train a model by using a data generator approach which only stores in memory the data required for that training batch.
Furthermore, efficient training over large models is illustrated through the training of a 7-layer UNet with input data dimensions of 4096,4096.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Deep learning applications are drastically progressing in seismic processing
and interpretation tasks. However, the majority of approaches subsample data
volumes and restrict model sizes to minimise computational requirements.
Subsampling the data risks losing vital spatio-temporal information which could
aid training whilst restricting model sizes can impact model performance, or in
some extreme cases, renders more complicated tasks such as segmentation
impossible. This paper illustrates how to tackle the two main issues of
training of large neural networks: memory limitations and impracticably large
training times. Typically, training data is preloaded into memory prior to
training, a particular challenge for seismic applications where data is
typically four times larger than that used for standard image processing tasks
(float32 vs. uint8). Using a microseismic use case, we illustrate how over
750GB of data can be used to train a model by using a data generator approach
which only stores in memory the data required for that training batch.
Furthermore, efficient training over large models is illustrated through the
training of a 7-layer UNet with input data dimensions of 4096X4096. Through a
batch-splitting distributed training approach, training times are reduced by a
factor of four. The combination of data generators and distributed training
removes any necessity of data 1 subsampling or restriction of neural network
sizes, offering the opportunity of utilisation of larger networks,
higher-resolution input data or moving from 2D to 3D problem spaces.
Related papers
- Partitioned Neural Network Training via Synthetic Intermediate Labels [0.0]
GPU memory constraints have become a notable bottleneck in training such sizable models.
This study advocates partitioning the model across GPU and generating synthetic intermediate labels to train individual segments.
This approach results in a more efficient training process that minimizes data communication while maintaining model accuracy.
arXiv Detail & Related papers (2024-03-17T13:06:29Z) - Deep Internal Learning: Deep Learning from a Single Input [88.59966585422914]
In many cases there is value in training a network just from the input at hand.
This is particularly relevant in many signal and image processing problems where training data is scarce and diversity is large.
This survey paper aims at covering deep internal-learning techniques that have been proposed in the past few years for these two important directions.
arXiv Detail & Related papers (2023-12-12T16:48:53Z) - Towards Federated Learning Under Resource Constraints via Layer-wise
Training and Depth Dropout [33.308067180286045]
Federated learning can be difficult to scale to large models when clients have limited resources.
We introduce Federated Layer-wise Learning to simultaneously reduce per-client memory, computation, and communication costs.
We also introduce Federated Depth Dropout, a complementary technique that randomly drops frozen layers during training, to further reduce resource usage.
arXiv Detail & Related papers (2023-09-11T03:17:45Z) - Dataset Quantization [72.61936019738076]
We present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets.
DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio.
arXiv Detail & Related papers (2023-08-21T07:24:29Z) - Rediscovering Hashed Random Projections for Efficient Quantization of
Contextualized Sentence Embeddings [113.38884267189871]
Training and inference on edge devices often requires an efficient setup due to computational limitations.
Pre-computing data representations and caching them on a server can mitigate extensive edge device computation.
We propose a simple, yet effective approach that uses randomly hyperplane projections.
We show that the embeddings remain effective for training models across various English and German sentence classification tasks that retain 94%--99% of their floating-point.
arXiv Detail & Related papers (2023-03-13T10:53:00Z) - Reconstructing Training Data from Model Gradient, Provably [68.21082086264555]
We reconstruct the training samples from a single gradient query at a randomly chosen parameter value.
As a provable attack that reveals sensitive training data, our findings suggest potential severe threats to privacy.
arXiv Detail & Related papers (2022-12-07T15:32:22Z) - DANCE: DAta-Network Co-optimization for Efficient Segmentation Model
Training and Inference [85.02494022662505]
DANCE is an automated simultaneous data-network co-optimization for efficient segmentation model training and inference.
It integrates automated data slimming which adaptively downsamples/drops input images and controls their corresponding contribution to the training loss guided by the images' spatial complexity.
Experiments and ablating studies demonstrate that DANCE can achieve "all-win" towards efficient segmentation.
arXiv Detail & Related papers (2021-07-16T04:58:58Z) - Data optimization for large batch distributed training of deep neural
networks [0.19336815376402716]
Current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale.
We propose a data optimization approach that utilize machine learning to implicitly smooth out the loss landscape resulting in fewer local minima.
Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.
arXiv Detail & Related papers (2020-12-16T21:22:02Z) - Predicting Training Time Without Training [120.92623395389255]
We tackle the problem of predicting the number of optimization steps that a pre-trained deep network needs to converge to a given value of the loss function.
We leverage the fact that the training dynamics of a deep network during fine-tuning are well approximated by those of a linearized model.
We are able to predict the time it takes to fine-tune a model to a given loss without having to perform any training.
arXiv Detail & Related papers (2020-08-28T04:29:54Z) - Approaches of large-scale images recognition with more than 50,000 categoris [0.0]
In this paper, we provide a viable solution for classifying large-scale species datasets using traditional CV techniques.
We use a dataset with more than 50, 000 categories, and all operations are done on common computer with l 6GB RAM and a CPU of 3. OGHz.
arXiv Detail & Related papers (2020-07-26T07:33:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.