An Overview of the Data-Loader Landscape: Comparative Performance
Analysis
- URL: http://arxiv.org/abs/2209.13705v1
- Date: Tue, 27 Sep 2022 21:40:56 GMT
- Title: An Overview of the Data-Loader Landscape: Comparative Performance
Analysis
- Authors: Iason Ofeidis, Diego Kiedanski, Leandros Tassiulas
- Abstract summary: Dataloaders might hold the key to drastically improving the performance of training jobs.
Recent advances have shown promise not only by considerably decreasing training time but also by offering new features such as loading data from remote storage like S3.
- Score: 6.913175606212201
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Dataloaders, in charge of moving data from storage into GPUs while training
machine learning models, might hold the key to drastically improving the
performance of training jobs. Recent advances have shown promise not only by
considerably decreasing training time but also by offering new features such as
loading data from remote storage like S3. In this paper, we are the first to
distinguish the dataloader as a separate component in the Deep Learning (DL)
workflow and to outline its structure and features. Finally, we offer a
comprehensive comparison of the different dataloading libraries available,
their trade-offs in terms of functionality, usability, and performance and the
insights derived from them.
Related papers
- Augment Before Copy-Paste: Data and Memory Efficiency-Oriented Instance Segmentation Framework for Sport-scenes [7.765333471208582]
In Visual Inductive Priors challenge (VIPriors2023), participants must train a model capable of precisely locating individuals on a basketball court.
We propose memory effIciency inStance framework based on visual inductive prior flow propagation.
Experiments demonstrate our model promising performance even under limited data and memory constraints.
arXiv Detail & Related papers (2024-03-18T08:44:40Z) - A Simple and Efficient Baseline for Data Attribution on Images [107.12337511216228]
Current state-of-the-art approaches require a large ensemble of as many as 300,000 models to accurately attribute model predictions.
In this work, we focus on a minimalist baseline, utilizing the feature space of a backbone pretrained via self-supervised learning to perform data attribution.
Our method is model-agnostic and scales easily to large datasets.
arXiv Detail & Related papers (2023-11-03T17:29:46Z) - SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations [76.45009891152178]
Pretraining-finetuning approach can alleviate the labeling burden by fine-tuning a pre-trained backbone across various downstream datasets as well as tasks.
We show, for the first time, that general representations learning can be achieved through the task of occupancy prediction.
Our findings will facilitate the understanding of LiDAR points and pave the way for future advancements in LiDAR pre-training.
arXiv Detail & Related papers (2023-09-19T11:13:01Z) - Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training [44.790636524264]
Point Prompt Training is a novel framework for multi-dataset synergistic learning in the context of 3D representation learning.
It can overcome the negative transfer associated with synergistic learning and produce generalizable representations.
It achieves state-of-the-art performance on each dataset using a single weight-shared model with supervised multi-dataset training.
arXiv Detail & Related papers (2023-08-18T17:59:57Z) - ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
We introduce Action-Aware Embodied Learning for Perception (ALP)
ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective.
We show that ALP outperforms existing baselines in several downstream perception tasks.
arXiv Detail & Related papers (2023-06-16T21:51:04Z) - Profiling and Improving the PyTorch Dataloader for high-latency Storage:
A Technical Report [0.7349727826230862]
This work focuses on the data loading pipeline in the PyTorch Framework.
We show that for classification tasks that involve loading many files, like images, the training wall-time can be significantly improved.
With our new, modified ConcurrentDataloader we can reach improvements in GPU utilization and significantly reduce batch loading time, up to 12X.
arXiv Detail & Related papers (2022-11-09T14:16:30Z) - A Data-Based Perspective on Transfer Learning [76.30206800557411]
We take a closer look at the role of the source dataset's composition in transfer learning.
Our framework gives rise to new capabilities such as pinpointing transfer learning brittleness.
arXiv Detail & Related papers (2022-07-12T17:58:28Z) - Knowledge Distillation as Efficient Pre-training: Faster Convergence,
Higher Data-efficiency, and Better Transferability [53.27240222619834]
Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks.
Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
arXiv Detail & Related papers (2022-03-10T06:23:41Z) - COLA: COarse LAbel pre-training for 3D semantic segmentation of sparse
LiDAR datasets [3.8243923744440926]
Transfer learning is a proven technique in 2D computer vision to leverage the large amount of data available and achieve high performance.
In this work, we tackle the case of real-time 3D semantic segmentation of sparse autonomous driving LiDAR scans.
We introduce a new pre-training task: coarse label pre-training, also called COLA.
arXiv Detail & Related papers (2022-02-14T17:19:23Z) - Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training.
We experimentally verify that the new dataset can significantly improve the ability of the learned FER model.
To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
arXiv Detail & Related papers (2020-05-18T09:36:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.