Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks

• URL: http://arxiv.org/abs/2304.08925v1
• Date: Tue, 18 Apr 2023 11:57:38 GMT
• Title: Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks
• Authors: Ping Gong, Yuxin Ma, Cheng Li, Xiaosong Ma, Sam H. Noh
• Abstract summary: We run experiments to test the performance implications of the two major data preprocessing methods using either raw data or record files. We identify the potential causes, exercise a variety of optimization methods, and present their pros and cons.
• Score: 8.977436072381973
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we primarily focus on understanding the data preprocessing pipeline for DNN Training in the public cloud. First, we run experiments to test the performance implications of the two major data preprocessing methods using either raw data or record files. The preliminary results show that data preprocessing is a clear bottleneck, even with the most efficient software and hardware configuration enabled by NVIDIA DALI, a high-optimized data preprocessing library. Second, we identify the potential causes, exercise a variety of optimization methods, and present their pros and cons. We hope this work will shed light on the new co-design of ``data storage, loading pipeline'' and ``training framework'' and flexible resource configurations between them so that the resources can be fully exploited and performance can be maximized.

Related papers

• DCP: Learning Accelerator Dataflow for Neural Network via Propagation [52.06154296196845]
  This work proposes an efficient data-centric approach, named Dataflow Code Propagation (DCP), to automatically find the optimal dataflow for DNN layers in seconds without human effort. DCP learns a neural predictor to efficiently update the dataflow codes towards the desired gradient directions to minimize various optimization objectives. For example, without using additional training data, DCP surpasses the GAMMA method that performs a full search using thousands of samples.
  arXiv  Detail & Related papers  (2024-10-09T05:16:44Z)
• An Integrated Data Processing Framework for Pretraining Foundation Models [57.47845148721817]
  Researchers and practitioners often have to manually curate datasets from difference sources. We propose a data processing framework that integrates a Processing Module and an Analyzing Module. The proposed framework is easy to use and highly flexible.
  arXiv  Detail & Related papers  (2024-02-26T07:22:51Z)
• Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP [43.7219097444333]
  We introduce a testbed of six publicly available data sources to investigate how pre-training distributions induce robustness in CLIP. We find that the performance of the pre-training data varies substantially across distribution shifts. We find that combining multiple sources does not necessarily yield better models, but rather dilutes the robustness of the best individual data source.
  arXiv  Detail & Related papers  (2022-08-10T18:24:23Z)
• Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability [53.27240222619834]
  Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks. Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
  arXiv  Detail & Related papers  (2022-03-10T06:23:41Z)
• Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines [77.45213180689952]
  Preprocessing pipelines in deep learning aim to provide sufficient data throughput to keep the training processes busy. We introduce a new perspective on efficiently preparing datasets for end-to-end deep learning pipelines. We obtain an increased throughput of 3x to 13x compared to an untuned system.
  arXiv  Detail & Related papers  (2022-02-17T14:31:58Z)
• Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice [52.11183787786718]
  Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications. Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy. We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
  arXiv  Detail & Related papers  (2021-11-24T06:18:32Z)
• Evaluation of Load Prediction Techniques for Distributed Stream Processing [0.0]
  Distributed Stream Processing (DSP) systems enable processing large streams of continuous data to produce results in near to real time. The rate at which events arrive at DSP systems can vary considerably over time. A priori knowledge of incoming workloads enables proactive approaches to resource management and optimization.
  arXiv  Detail & Related papers  (2021-08-10T15:25:32Z)
• Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics [24.62486707803304]
  In this work, we examine end-to-end DNN execution in visual analytics systems on modern accelerators. To address the bottleneck of preprocessing, we introduce two optimizations for end-to-end visual analytics systems. We show that its optimizations can achieve up to 5.9x end-to-end throughput improvements at a fixed accuracy over recent work in visual analytics.
  arXiv  Detail & Related papers  (2020-07-25T20:26:05Z)
• Analyzing and Mitigating Data Stalls in DNN Training [7.444113272493349]
  We present the first comprehensive analysis of how the input data pipeline affects the training time of Deep Neural Networks (DNNs) We find that in many cases, DNN training time is dominated by data stall time: time spent waiting for data to be fetched and preprocessed. We implement three simple but effective techniques in a data-loading library, CoorDL, to mitigate data stalls.
  arXiv  Detail & Related papers  (2020-07-14T02:16:56Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.