Scaling Vision Transformers: Evaluating DeepSpeed for Image-Centric Workloads

• URL: http://arxiv.org/abs/2602.21081v1
• Date: Tue, 24 Feb 2026 16:45:12 GMT
• Title: Scaling Vision Transformers: Evaluating DeepSpeed for Image-Centric Workloads
• Authors: Huy Trinh, Rebecca Ma, Zeqi Yu, Tahsin Reza,
• Abstract summary: Vision Transformers (ViTs) have demonstrated remarkable potential in image processing tasks by utilizing self-attention mechanisms to capture global relationships within data.<n>This study aims to leverage DeepSpeed, a highly efficient distributed training framework, to enhance the scalability and performance of ViTs.
• Score: 0.1679937788852768
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Vision Transformers (ViTs) have demonstrated remarkable potential in image processing tasks by utilizing self-attention mechanisms to capture global relationships within data. However, their scalability is hindered by significant computational and memory demands, especially for large-scale models with many parameters. This study aims to leverage DeepSpeed, a highly efficient distributed training framework that is commonly used for language models, to enhance the scalability and performance of ViTs. We evaluate intra- and inter-node training efficiency across multiple GPU configurations on various datasets like CIFAR-10 and CIFAR-100, exploring the impact of distributed data parallelism on training speed, communication overhead, and overall scalability (strong and weak scaling). By systematically varying software parameters, such as batch size and gradient accumulation, we identify key factors influencing performance of distributed training. The experiments in this study provide a foundational basis for applying DeepSpeed to image-related tasks. Future work will extend these investigations to deepen our understanding of DeepSpeed's limitations and explore strategies for optimizing distributed training pipelines for Vision Transformers.

Related papers

• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective [6.51239603014107]
  Large Language Models (LLMs) have pushed training workloads beyond the limits of single-node analysis.<n>We present a comprehensive characterization of LLM training across diverse real-world workloads and hardware platforms.
  arXiv  Detail & Related papers  (2025-09-12T16:05:07Z)
• Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategies [66.83950068218033]
  Scaling Laws demonstrate that scaling model parameters and training data enhances learning performance.<n>Despite its potential to improve performance, the integration of scaling laws into deep reinforcement learning has not been fully realized.<n>This review addresses this gap by systematically analyzing scaling strategies in three dimensions: data, network, and training budget.
  arXiv  Detail & Related papers  (2025-08-05T08:03:12Z)
• Scaling Intelligence: Designing Data Centers for Next-Gen Language Models [0.6168147650666682]
  Large Language Models (LLMs), such as GPT-4 with 1.8 trillion parameters, demand a fundamental rethinking of data center architecture.<n>Our work provides a comprehensive co-design framework that jointly explores FLOPS, bandwidth and capacity, multiple network topologies.<n>We quantify the benefits of overlapping compute and communication, leveraging hardware-accelerated collectives, widening the scale-out domain, and increasing memory capacity.
  arXiv  Detail & Related papers  (2025-06-17T22:29:37Z)
• Underlying Semantic Diffusion for Effective and Efficient In-Context Learning [113.4003355229632]
  Underlying Semantic Diffusion (US-Diffusion) is an enhanced diffusion model that boosts underlying semantics learning, computational efficiency, and in-context learning capabilities.<n>We present a Feedback-Aided Learning (FAL) framework, which leverages feedback signals to guide the model in capturing semantic details.<n>We also propose a plug-and-play Efficient Sampling Strategy (ESS) for dense sampling at time steps with high-noise levels.
  arXiv  Detail & Related papers  (2025-03-06T03:06:22Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Optimizing Vision Transformers with Data-Free Knowledge Transfer [8.323741354066474]
  Vision transformers (ViTs) have excelled in various computer vision tasks due to their superior ability to capture long-distance dependencies. We propose compressing large ViT models using Knowledge Distillation (KD), which is implemented data-free to circumvent limitations related to data availability.
  arXiv  Detail & Related papers  (2024-08-12T07:03:35Z)
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning [29.798780069556074]
  Federated learning (FL) has emerged as a promising approach to collaboratively train machine learning models across multiple edge devices. We propose OnDev-LCT: Lightweight Convolutional Transformers for On-Device vision tasks with limited training data and resources.
  arXiv  Detail & Related papers  (2024-01-22T02:17:36Z)
• ViR: Towards Efficient Vision Retention Backbones [97.93707844681893]
  We propose a new class of computer vision models, dubbed Vision Retention Networks (ViR) ViR has dual parallel and recurrent formulations, which strike an optimal balance between fast inference and parallel training with competitive performance. We have validated the effectiveness of ViR through extensive experiments with different dataset sizes and various image resolutions.
  arXiv  Detail & Related papers  (2023-10-30T16:55:50Z)
• Controllable Data Augmentation Through Deep Relighting [75.96144853354362]
  We explore how to augment a varied set of image datasets through relighting so as to improve the ability of existing models to be invariant to illumination changes. We develop a tool, based on an encoder-decoder network, that is able to quickly generate multiple variations of the illumination of various input scenes. We demonstrate that by training models on datasets that have been augmented with our pipeline, it is possible to achieve higher performance on localization benchmarks.
  arXiv  Detail & Related papers  (2021-10-26T20:02:51Z)
• Deflating Dataset Bias Using Synthetic Data Augmentation [8.509201763744246]
  State-of-the-art methods for most vision tasks for Autonomous Vehicles (AVs) rely on supervised learning. The goal of this paper is to investigate the use of targeted synthetic data augmentation for filling gaps in real datasets for vision tasks. Empirical studies on three different computer vision tasks of practical use to AVs consistently show that having synthetic data in the training mix provides a significant boost in cross-dataset generalization performance.
  arXiv  Detail & Related papers  (2020-04-28T21:56:10Z)
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training [126.49572353148262]
  We study two factors in neural network training: data parallelism and sparsity. Despite their promising benefits, understanding of their effects on neural network training remains elusive.
  arXiv  Detail & Related papers  (2020-03-25T10:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.