Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

• URL: http://arxiv.org/abs/2607.08511v1
• Date: Thu, 09 Jul 2026 14:06:39 GMT
• Title: Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures
• Abstract summary: We investigate the impact of schedulers on classification accuracy across a diverse pool of architectures.<n>We applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10.<n>The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies.
• Score: 48.83701310501069
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While classical AutoML approaches often treat the scheduler as a secondary hyperparameter, we systematically investigate its impact on classification accuracy across a diverse pool of architectures. We evaluated 30 representative architectures from convolutional and transformer families within the LEMUR neural network dataset. Through automated source-code injection, we applied 25 scheduler configurations across nine PyTorch families, evaluating a total of 3,938 model variants on CIFAR-10. Our best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%. The results show that the choice of scheduler depends heavily on the architecture: CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies. The resulting accuracy landscape, contributed to the LEMUR nn-dataset, provides a practical reference for principled scheduler selection.

Related papers

• Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis [48.83701310501069]
  We introduce a MinHash-based similarity scheduling framework that constructs a progressive curriculum over neural architecture code for neural architecture search (NAS)<n>We evaluate the framework on OlympicCoder-7B within the LEMUR benchmark on CIFAR-10 image classification.<n>We report a cross-dataset transfer observation on SVHN, where direct base-model generation without curriculum yields 27% peak SR at substantially lower accuracy (60.5%) than the CIFAR-10 equivalent.
  arXiv  Detail & Related papers  (2026-07-13T14:14:30Z)
• From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT [48.83701310501069]
  We present a classification task integrated into the NNGPT framework.<n>A fine-tuned LLM predicts which of two image classification datasets a given neural network architecture achieves higher accuracy on.<n>The results establish that LLMs can be fine-tuned to predict cross-dataset suitability from neural network code.
  arXiv  Detail & Related papers  (2026-05-05T12:30:19Z)
• Empirical Ablation and Ensemble Optimization of a Convolutional Neural Network for CIFAR-10 Classification [0.0]
  Convolutional neural networks (CNNs) remain a central approach in image classification.<n>This paper presents an empirical ablation-based study of CNN optimization for the CIFAR-10 benchmark.
  arXiv  Detail & Related papers  (2026-04-26T20:08:26Z)
• Resonant Sparse Geometry Networks [0.0]
  We introduce Resonant Sparse Geometry Networks (RSGN), a brain-inspired architecture with self-organizing sparse hierarchical input-dependent connectivity.<n> RSGN embeds computational nodes in learned hyperbolic space where connection strength decays with geodesic distance, achieving dynamic sparsity that adapts to each input.
  arXiv  Detail & Related papers  (2026-01-26T01:45:51Z)
• SymRAG: Efficient Neuro-Symbolic Retrieval Through Adaptive Query Routing [8.775121469887033]
  Current Retrieval-Augmented Generation systems use uniform processing, causing inefficiency as simple queries consume resources similar to complex multi-hop tasks.<n>We present SymRAG, a framework that introduces adaptive query routing via real-time complexity and load assessment to select symbolic, neural, or hybrid pathways.
  arXiv  Detail & Related papers  (2025-06-15T22:35:43Z)
• Meta knowledge assisted Evolutionary Neural Architecture Search [38.55611683982936]
  This paper introduces an efficient EC-based NAS method to solve problems via an innovative meta-learning framework.<n>An adaptive surrogate model is designed through an adaptive threshold to select the potential architectures.<n> Experiments on CIFAR-10, CIFAR-100, and ImageNet1K datasets demonstrate that the proposed method achieves high performance comparable to that of many state-of-the-art peer methods.
  arXiv  Detail & Related papers  (2025-04-30T11:43:07Z)
• End-to-End Implicit Neural Representations for Classification [57.55927378696826]
  Implicit neural representations (INRs) encode a signal in neural network parameters and show excellent results for signal reconstruction.<n>INR-based classification still significantly under-performs compared to pixel-based methods like CNNs.<n>This work presents an end-to-end strategy for initializing SIRENs together with a learned learning-rate scheme.
  arXiv  Detail & Related papers  (2025-03-23T16:02:23Z)
• RSQ: Learning from Important Tokens Leads to Better Quantized LLMs [65.5558181902098]
  Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining.<n>We propose RSQ (Rotate, Scale, then Quantize), which applies rotations to the model to mitigate outliers.<n>We demonstrate that RSQ consistently outperforms baseline methods across multiple downstream tasks and three model families.
  arXiv  Detail & Related papers  (2025-03-03T18:46:33Z)
• Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction [3.3810628880631226]
  This paper systematically investigates the combination of log data embedding strategies and Deep Learning (DL) types for failure prediction. To that end, we propose a modular architecture to accommodate various configurations of embedding strategies and DL-based encoders. Using the F1 score metric, our results show that the best overall performing configuration is a CNN-based encoder with Logkey2vec.
  arXiv  Detail & Related papers  (2023-03-13T16:04:14Z)
• ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models [56.21470608621633]
  We propose a time estimation framework to decouple the architectural search from the target hardware. The proposed methodology extracts a set of models from micro- kernel and multi-layer benchmarks and generates a stacked model for mapping and network execution time estimation. We compare estimation accuracy and fidelity of the generated mixed models, statistical models with the roofline model, and a refined roofline model for evaluation.
  arXiv  Detail & Related papers  (2021-05-07T11:39:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.