One Size Does Not Fit All: Architecture-Aware Adaptive Batch Scheduling with DEBA

• URL: http://arxiv.org/abs/2511.03809v1
• Date: Wed, 05 Nov 2025 19:19:39 GMT
• Title: One Size Does Not Fit All: Architecture-Aware Adaptive Batch Scheduling with DEBA
• Abstract summary: We introduce DEBA, an adaptive batch scheduler that monitors gradient variance, gradient norm variation and loss variation.<n>We demonstrate that the architecture fundamentally determines adaptation efficacy.
• Score: 9.344832445480415
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adaptive batch size methods aim to accelerate neural network training, but existing approaches apply identical adaptation strategies across all architectures, assuming a one-size-fits-all solution. We introduce DEBA (Dynamic Efficient Batch Adaptation), an adaptive batch scheduler that monitors gradient variance, gradient norm variation and loss variation to guide batch size adaptations. Through systematic evaluation across six architectures (ResNet-18/50, DenseNet-121, EfficientNet-B0, MobileNet-V3, ViT-B16) on CIFAR-10 and CIFAR-100, with five random seeds per configuration, we demonstrate that the architecture fundamentally determines adaptation efficacy. Our findings reveal that: (1) lightweight and medium-depth architectures (MobileNet-V3, DenseNet-121, EfficientNet-B0) achieve a 45-62% training speedup with simultaneous accuracy improvements of 1-7%; (2) shallow residual networks (ResNet-18) show consistent gains of +2.4 - 4.0% in accuracy, 36 - 43% in speedup, while deep residual networks (ResNet-50) exhibit high variance and occasional degradation; (3) already-stable architectures (ViT-B16) show minimal speedup (6%) despite maintaining accuracy, indicating that adaptation benefits vary with baseline optimization characteristics. We introduce a baseline characterization framework using gradient stability metrics (stability score, gradient norm variation) that predicts which architectures will benefit from adaptive scheduling. Our ablation studies reveal critical design choices often overlooked in prior work: sliding window statistics (vs. full history) and sufficient cooldown periods (5+ epochs) between adaptations are essential for success. This work challenges the prevailing assumption that adaptive methods generalize across architectures and provides the first systematic evidence that batch size adaptation requires an architecture-aware design.

Related papers

• Model-Free Surrogate-Assisted Neural Architecture Search for Evolving Variable-Length Dense Blocks [2.473241744699909]
  We propose a Model-Free Surrogate PSO Network (MFSPNet) for evolving convolutional neural network architectures.<n>MFSPNet integrates a lightweight model-free surrogate predictor within a particle swarm optimization (PSO) framework, eliminating the need for pre-trained surrogate models.<n>Under a consistent training protocol with ten independent runs, the proposed method attains error rates of 3.91%, 17.68%, and 1.91% on CIFAR-10, CIFAR-100, and SVHN, respectively.
  arXiv  Detail & Related papers  (2026-09-02T11:32:07Z)
• Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures [48.83701310501069]
  We investigate the impact of schedulers on classification accuracy across a diverse pool of architectures.<n>We applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10.<n>The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies.
  arXiv  Detail & Related papers  (2026-07-09T14:06:39Z)
• Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study [48.83701310501069]
  This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models.<n>We conduct a systematic empirical study of all $3 times 6 = 18$ combinations of six classifiers paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets.<n>Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across
  arXiv  Detail & Related papers  (2026-06-18T20:51:42Z)
• Empirical Ablation and Ensemble Optimization of a Convolutional Neural Network for CIFAR-10 Classification [0.0]
  Convolutional neural networks (CNNs) remain a central approach in image classification.<n>This paper presents an empirical ablation-based study of CNN optimization for the CIFAR-10 benchmark.
  arXiv  Detail & Related papers  (2026-04-26T20:08:26Z)
• NOVAK: Unified adaptive optimizer for deep neural networks [0.0]
  NOVAK is a gradient-based optimization algorithm that integrates adaptive moment estimation, rectified learning-rate scheduling, decoupled weight regularization, multiple variants of Nesterov momentum, and lookahead synchronization into a unified, performance-oriented framework.
  arXiv  Detail & Related papers  (2026-01-11T13:03:57Z)
• Modified TSception for Analyzing Driver Drowsiness and Mental Workload from EEG [6.767263284839525]
  Driver drowsiness remains a primary cause of traffic accidents, necessitating the development of real-time, reliable detection systems.<n>This study presents a Modified TSception architecture designed for the robust assessment of driver fatigue using Electroencephalography (EEG)<n>The architecture's generalizability is validated on the STEW mental workload dataset.
  arXiv  Detail & Related papers  (2025-12-25T17:48:11Z)
• Progressive Supernet Training for Efficient Visual Autoregressive Modeling [56.15415456746672]
  We propose a training strategy that breaks through the frontier of generation quality for both paradigms and the full network.<n>Experiments on ImageNet demonstrate that, compared to the pretrained VAR-d30, VARiant-d16 and VARiant-d8 achieve nearly equivalent quality.<n> VARiant-d2 achieves 3.5 times speedup and 80% memory reduction at moderate quality cost.
  arXiv  Detail & Related papers  (2025-11-20T16:59:24Z)
• Tri-Accel: Curvature-Aware Precision-Adaptive and Memory-Elastic Optimization for Efficient GPU Usage [0.6511750267058007]
  Tri-Accel is a unified optimization framework that co-adapts three acceleration strategies along with adaptive parameters during training.<n>On CIFAR-10 with ResNet-18 and EfficientNet-B0, Tri-Accel achieves up to 9.9% reduction in training time and 13.3% lower memory usage.<n>Compared to static mixed-precision training, Tri-Accel maintains 78.1% accuracy while reducing memory footprint from 0.35GB to 0.31GB on standard hardware.
  arXiv  Detail & Related papers  (2025-08-23T05:38:42Z)
• Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Acceleration [1.3225694028747144]
  This paper presents an efficient Loss-Aware Automatic Selection of Structured Pruning Criteria (LAASP) for slimming and accelerating deep neural networks.<n>The pruning-while-training approach eliminates the first stage and integrates the second and third stages into a single cycle.<n>Experiments on the VGGNet and ResNet models on the CIFAR-10 and ImageNet benchmark datasets demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2025-06-25T06:18:46Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks [48.089501687522954]
  We propose a novel layer-adaptive weight-pruning approach for Deep Neural Networks (DNNs) Our approach takes into account the collective influence of all layers to design a layer-adaptive pruning scheme. Our experiments demonstrate the superiority of our approach over existing methods on the ImageNet and CIFAR-10 datasets.
  arXiv  Detail & Related papers  (2023-08-21T03:22:47Z)
• SmoothNets: Optimizing CNN architecture design for differentially private deep learning [69.10072367807095]
  DPSGD requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. We distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training.
  arXiv  Detail & Related papers  (2022-05-09T07:51:54Z)
• Inception Convolution with Efficient Dilation Search [121.41030859447487]
  Dilation convolution is a critical mutant of standard convolution neural network to control effective receptive fields and handle large scale variance of objects. We propose a new mutant of dilated convolution, namely inception (dilated) convolution where the convolutions have independent dilation among different axes, channels and layers. We explore a practical method for fitting the complex inception convolution to the data, a simple while effective dilation search algorithm(EDO) based on statistical optimization is developed.
  arXiv  Detail & Related papers  (2020-12-25T14:58:35Z)
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining [65.39532971991778]
  We present an accuracy predictor that scores architecture and training recipes jointly, guiding both sample selection and ranking. We run fast evolutionary searches in just CPU minutes to generate architecture-recipe pairs for a variety of resource constraints. FBNetV3 makes up a family of state-of-the-art compact neural networks that outperform both automatically and manually-designed competitors.
  arXiv  Detail & Related papers  (2020-06-03T05:20:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.