Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis

• URL: http://arxiv.org/abs/2508.16550v1
• Date: Fri, 22 Aug 2025 17:16:06 GMT
• Title: Enhanced NIRMAL Optimizer With Damped Nesterov Acceleration: A Comparative Analysis
• Abstract summary: Enhanced NIRMAL (Novel Integrated Robust Multi-Adaptation with Damped Learning Nesterov Acceleration)<n>We evaluate Enhanced NIRMAL against Adam, SGD with Momentum, Nesterov, and the original NIRMAL on four benchmark image classification datasets.<n> Enhanced NIRMAL achieves a test accuracy of 46.06% and the lowest test loss (1.960435) on CIFAR-100.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study introduces the Enhanced NIRMAL (Novel Integrated Robust Multi-Adaptation Learning with Damped Nesterov Acceleration) optimizer, an improved version of the original NIRMAL optimizer. By incorporating an $(\alpha, r)$-damped Nesterov acceleration mechanism, Enhanced NIRMAL improves convergence stability while retaining chess-inspired strategies of gradient descent, momentum, stochastic perturbations, adaptive learning rates, and non-linear transformations. We evaluate Enhanced NIRMAL against Adam, SGD with Momentum, Nesterov, and the original NIRMAL on four benchmark image classification datasets: MNIST, FashionMNIST, CIFAR-10, and CIFAR-100, using tailored convolutional neural network (CNN) architectures. Enhanced NIRMAL achieves a test accuracy of 46.06\% and the lowest test loss (1.960435) on CIFAR-100, surpassing the original NIRMAL (44.34\% accuracy) and closely rivaling SGD with Momentum (46.43\% accuracy). These results underscore Enhanced NIRMAL's superior generalization and stability, particularly on complex datasets.

Related papers

• Reducing ANN-SNN Conversion Error via Residual Membrane Potential Alignment [52.910770415431244]
  Spiking Neural Networks (SNNs) serve as core architectures for neuromorphic computing.<n>ANN-SNN conversion circumvents issues by reusing well-trained ANN weights for low-latency, energy-efficient inference.<n>Existing conversion schemes suffer from severe accuracy drops at small timesteps, large inference delays and cumulative quantization errors.
  arXiv  Detail & Related papers  (2026-08-14T04:46:03Z)
• ISO: An RLVR-Native Optimization Stack [50.40395312897848]
  We operationalize spectral inheritance as Isospectral Optimization (ISO)<n>Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model.<n>Online, ISO-Merger applies a chosen base, including AdamW and Muon, to the frame variables while keeping the base spectra fixed.
  arXiv  Detail & Related papers  (2026-07-21T17:51:36Z)
• Constraint-Aware Optimization for Robust Protein Stability Prediction [0.3316543849024811]
  Multimodal $G$ integrating protein language models with inverse-folding representations achieve strong in-distribution accuracy on the Megascale dataset.<n>Existing approaches address these limitations primarily through additional architectural components.<n>We introduce a constraint-aware optimization framework combining Balanced Mean Squared Error, a Siamese anti-symmetric regularizer, and a novel OOD-margin loss on the per-position predictors feature representation.
  arXiv  Detail & Related papers  (2026-06-06T11:04:52Z)
• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization [8.412194372218895]
  We introduce Singularity-aware Adam (S-Adam), a novel that stabilizes training by dynamically modulating step sizes based on local geometric instability.<n>S-Adam incorporates an adaptive damping mechanism that decelerates updates in high-instability regions while preserving fast convergence in smooth basins.<n> Empirical evaluations on optimal Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD.
  arXiv  Detail & Related papers  (2026-05-28T08:00:40Z)
• Weak-SIGReg: Covariance Regularization for Stable Deep Learning [0.0]
  This work adopts Sketched Isotropic Gaussian Regularization ( SIGReg), recently introduced in the LeJEPA self-supervised framework, and repurposes it as a general optimization stabilizer for supervised learning.<n>Inspired by interacting particle systems, representation collapse is viewed as drift; SIGReg constrains the representation density towards an isotropic Gaussian, mitigating this drift.<n> Empirically, SIGReg recovers the training of a ViT on CIFAR-100 from a collapsed 20.73% to 72.02% accuracy without architectural hacks and significantly improves the convergence of deep vanillas trained with pure SGD. Code is available at
  arXiv  Detail & Related papers  (2026-03-06T05:32:40Z)
• NOVAK: Unified adaptive optimizer for deep neural networks [0.0]
  NOVAK is a gradient-based optimization algorithm that integrates adaptive moment estimation, rectified learning-rate scheduling, decoupled weight regularization, multiple variants of Nesterov momentum, and lookahead synchronization into a unified, performance-oriented framework.
  arXiv  Detail & Related papers  (2026-01-11T13:03:57Z)
• Randomness and Interpolation Improve Gradient Descent [1.7551939488475077]
  The paper introduces two Avolutions, named Interpolational Gradient Descent (IAGD) and Noise-Regularized Gradient Descent (NRSGD)<n>IAGD uses Newton Interpolation to expedite the convergence process during training, assuming relevancy in gradients between iterations.<n>NRSGD incorporates a noise regularization technique that controlled noise to the gradients during the optimization process.
  arXiv  Detail & Related papers  (2025-10-14T23:32:01Z)
• NIRVANA: Structured pruning reimagined for large language models compression [50.651730342011014]
  We introduce NIRVANA, a novel pruning method designed to balance immediate zero-shot preservation accuracy with robust fine-tuning.<n>To further address the unique challenges posed by structured pruning, NIRVANA incorporates an adaptive sparsity allocation mechanism across layers and modules.<n>Experiments conducted on Llama3, Qwen, T5 models demonstrate that NIRVANA outperforms existing structured pruning methods under equivalent sparsity constraints.
  arXiv  Detail & Related papers  (2025-09-17T17:59:00Z)
• Comparative Analysis of Novel NIRMAL Optimizer Against Adam and SGD with Momentum [0.8437187555622164]
  NIRMAL (Novel Integrated Robust Multi-Adaptation Learning) is a novel optimization algorithm that combines multiple strategies inspired by the movements of the chess piece.<n>NIRMAL achieves competitive performance, particularly on the more challenging CIFAR-100 dataset.<n>These findings underscore NIRMAL's significant ability as a versatile and effective datasets for various deep learning tasks.
  arXiv  Detail & Related papers  (2025-08-06T10:30:22Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [61.79405341803085]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in federated learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in federated learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• FMDConv: Fast Multi-Attention Dynamic Convolution via Speed-Accuracy Trade-off [12.900580256269155]
  We propose Fast Multi-Attention Dynamic Convolution (FMDConv), which integrates input attention, temperature-degraded kernel attention, and output attention to optimize the speed-accuracy trade-off.<n>Experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that FMDConv reduces the computational cost by up to 49.8% on ResNet-18 and 42.2% on ResNet-50.
  arXiv  Detail & Related papers  (2025-03-21T20:23:32Z)
• Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise [60.92029979853314]
  We investigate the roles of gradient normalization and clipping in ensuring the convergence of Gradient Descent (SGD) under heavy-tailed noise. Our work provides the first theoretical evidence demonstrating the benefits of gradient normalization in SGD under heavy-tailed noise. We introduce an accelerated SGD variant incorporating gradient normalization and clipping, further enhancing convergence rates under heavy-tailed noise.
  arXiv  Detail & Related papers  (2024-10-21T22:40:42Z)
• MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers [41.56951365163419]
  "MixedNUTS" is a training-free method where the output logits of a robust classifier are processed by nonlinear transformations with only three parameters. MixedNUTS then converts the transformed logits into probabilities and mixes them as the overall output. On CIFAR-10, CIFAR-100, and ImageNet datasets, experimental results with custom strong adaptive attacks demonstrate MixedNUTS's vastly improved accuracy and near-SOTA robustness.
  arXiv  Detail & Related papers  (2024-02-03T21:12:36Z)
• Semantic Perturbations with Normalizing Flows for Improved Generalization [62.998818375912506]
  We show that perturbations in the latent space can be used to define fully unsupervised data augmentations. We find that our latent adversarial perturbations adaptive to the classifier throughout its training are most effective.
  arXiv  Detail & Related papers  (2021-08-18T03:20:00Z)
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization [80.8226518642952]
  We study optimization of areas under precision-recall curves (AUPRC), which is widely used for imbalanced tasks. We develop novel momentum methods with a better iteration of $O (1/epsilon4)$ for finding an $epsilon$stationary solution. We also design a novel family of adaptive methods with the same complexity of $O (1/epsilon4)$, which enjoy faster convergence in practice.
  arXiv  Detail & Related papers  (2021-07-02T16:21:52Z)
• Fast Convergence of DETR with Spatially Modulated Co-Attention [83.19863907905666]
  We propose a simple yet effective scheme for improving the Detection Transformer framework, namely Spatially Modulated Co-Attention (SMCA) mechanism. Our proposed SMCA increases DETR's convergence speed by replacing the original co-attention mechanism in the decoder. Our fully-fledged SMCA can achieve better performance compared to DETR with a dilated convolution-based backbone.
  arXiv  Detail & Related papers  (2021-01-19T03:52:44Z)
• CASTLE: Regularization via Auxiliary Causal Graph Discovery [89.74800176981842]
  We introduce Causal Structure Learning (CASTLE) regularization and propose to regularize a neural network by jointly learning the causal relationships between variables. CASTLE efficiently reconstructs only the features in the causal DAG that have a causal neighbor, whereas reconstruction-based regularizers suboptimally reconstruct all input features.
  arXiv  Detail & Related papers  (2020-09-28T09:49:38Z)
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning [91.13797346047984]
  We introduce ADAHESSIAN, a second order optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates. We show that ADAHESSIAN achieves new state-of-the-art results by a large margin as compared to other adaptive optimization methods.
  arXiv  Detail & Related papers  (2020-06-01T05:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.