An Overview of the Data-Loader Landscape: Comparative Performance Analysis

• URL: http://arxiv.org/abs/2209.13705v1
• Date: Tue, 27 Sep 2022 21:40:56 GMT
• Title: An Overview of the Data-Loader Landscape: Comparative Performance Analysis
• Authors: Iason Ofeidis, Diego Kiedanski, Leandros Tassiulas
• Abstract summary: Dataloaders might hold the key to drastically improving the performance of training jobs. Recent advances have shown promise not only by considerably decreasing training time but also by offering new features such as loading data from remote storage like S3.
• Score: 6.913175606212201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dataloaders, in charge of moving data from storage into GPUs while training machine learning models, might hold the key to drastically improving the performance of training jobs. Recent advances have shown promise not only by considerably decreasing training time but also by offering new features such as loading data from remote storage like S3. In this paper, we are the first to distinguish the dataloader as a separate component in the Deep Learning (DL) workflow and to outline its structure and features. Finally, we offer a comprehensive comparison of the different dataloading libraries available, their trade-offs in terms of functionality, usability, and performance and the insights derived from them.

Related papers

• OVERLORD: Ultimate Scaling of DataLoader for Multi-Source Large Foundation Model Training [17.215899004049778]
  We present OVERLORD, an industrial-grade distributed data loading architecture with three innovations. OVERLORD achieves: (1) 4.5x end-to-end training throughput improvement; (2) a minimum 3.6x reduction in CPU memory usage.
  arXiv  Detail & Related papers  (2025-04-14T03:31:22Z)
• OmniBal: Towards Fast Instruct-tuning for Vision-Language Models via Omniverse Computation Balance [35.40320275366383]
  Large-scale 3D parallel training on vision-language instruct-tuning models leads to an imbalanced computation load across different devices. We rebalanced the computational loads from data, model, and memory perspectives to address this issue. Our method's efficacy and generalizability were further demonstrated across various models and datasets.
  arXiv  Detail & Related papers  (2024-07-30T12:02:58Z)
• Augment Before Copy-Paste: Data and Memory Efficiency-Oriented Instance Segmentation Framework for Sport-scenes [7.765333471208582]
  In Visual Inductive Priors challenge (VIPriors2023), participants must train a model capable of precisely locating individuals on a basketball court. We propose memory effIciency inStance framework based on visual inductive prior flow propagation. Experiments demonstrate our model promising performance even under limited data and memory constraints.
  arXiv  Detail & Related papers  (2024-03-18T08:44:40Z)
• Less is More: High-value Data Selection for Visual Instruction Tuning [127.38740043393527]
  We propose a high-value data selection approach TIVE, to eliminate redundancy within the visual instruction data and reduce the training cost. Our approach using only about 15% data can achieve comparable average performance to the full-data fine-tuned model across eight benchmarks.
  arXiv  Detail & Related papers  (2024-03-14T16:47:25Z)
• A Simple and Efficient Baseline for Data Attribution on Images [107.12337511216228]
  Current state-of-the-art approaches require a large ensemble of as many as 300,000 models to accurately attribute model predictions. In this work, we focus on a minimalist baseline, utilizing the feature space of a backbone pretrained via self-supervised learning to perform data attribution. Our method is model-agnostic and scales easily to large datasets.
  arXiv  Detail & Related papers  (2023-11-03T17:29:46Z)
• SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations [76.45009891152178]
  Pretraining-finetuning approach can alleviate the labeling burden by fine-tuning a pre-trained backbone across various downstream datasets as well as tasks. We show, for the first time, that general representations learning can be achieved through the task of occupancy prediction. Our findings will facilitate the understanding of LiDAR points and pave the way for future advancements in LiDAR pre-training.
  arXiv  Detail & Related papers  (2023-09-19T11:13:01Z)
• Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training [44.790636524264]
  Point Prompt Training is a novel framework for multi-dataset synergistic learning in the context of 3D representation learning. It can overcome the negative transfer associated with synergistic learning and produce generalizable representations. It achieves state-of-the-art performance on each dataset using a single weight-shared model with supervised multi-dataset training.
  arXiv  Detail & Related papers  (2023-08-18T17:59:57Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Profiling and Improving the PyTorch Dataloader for high-latency Storage: A Technical Report [0.7349727826230862]
  This work focuses on the data loading pipeline in the PyTorch Framework. We show that for classification tasks that involve loading many files, like images, the training wall-time can be significantly improved. With our new, modified ConcurrentDataloader we can reach improvements in GPU utilization and significantly reduce batch loading time, up to 12X.
  arXiv  Detail & Related papers  (2022-11-09T14:16:30Z)
• Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability [53.27240222619834]
  Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks. Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
  arXiv  Detail & Related papers  (2022-03-10T06:23:41Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.