Foxtsage vs. Adam: Revolution or Evolution in Optimization?

• URL: http://arxiv.org/abs/2412.17855v1
• Date: Fri, 20 Dec 2024 03:20:58 GMT
• Title: Foxtsage vs. Adam: Revolution or Evolution in Optimization?
• Abstract summary: This study introduces Foxtsage, a novel hybrid optimisation approach that integrates the Hybrid FOX-TSA with Gradient Descent for training Multi-Layer Perceptron models.<n> Experimental results demonstrate that Foxtsage achieves a 42.03% reduction in loss mean (Foxtsage: 9.508, Adam: 16.402) and a 42.19% improvement in loss standard deviation (Foxtsage: 20.86, Adam: 36.085)
• Score: 4.939986309170004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optimization techniques are pivotal in neural network training, shaping both predictive performance and convergence efficiency. This study introduces Foxtsage, a novel hybrid optimisation approach that integrates the Hybrid FOX-TSA with Stochastic Gradient Descent for training Multi-Layer Perceptron models. The proposed Foxtsage method is benchmarked against the widely adopted Adam optimizer across multiple standard datasets, focusing on key performance metrics such as training loss, accuracy, precision, recall, F1-score, and computational time. Experimental results demonstrate that Foxtsage achieves a 42.03% reduction in loss mean (Foxtsage: 9.508, Adam: 16.402) and a 42.19% improvement in loss standard deviation (Foxtsage: 20.86, Adam: 36.085), reflecting enhanced consistency and robustness. Modest improvements in accuracy mean (0.78%), precision mean (0.91%), recall mean (1.02%), and F1-score mean (0.89%) further underscore its predictive performance. However, these gains are accompanied by an increased computational cost, with a 330.87% rise in time mean (Foxtsage: 39.541 seconds, Adam: 9.177 seconds). By effectively combining the global search capabilities of FOX-TSA with the stability and adaptability of SGD, Foxtsage presents itself as a robust and viable alternative for neural network optimization tasks.

Related papers

• Do Newer Lightweight CNNs Perform Better Under Resource Constraints? A Controlled Multigenerational Study of Architecture, Initialization, Training Budget, and Efficiency [0.0]
  This study compares nine lightweight CNN model packages across CIFAR-10, CIFAR-100, and Tiny ImageNet.<n>We report top-1 accuracy, macro F1, top-5 accuracy, parameter count, FP32 storage, GMACs, batch-size-1 latency on an NVIDIA L4 and AMD 5 5500U CPU.
  arXiv  Detail & Related papers  (2026-07-02T10:14:07Z)
• Evolving Deep Learning Optimizers [0.0]
  We present a genetic algorithm framework for discovering deep learning optimization algorithms.<n>Our approach encodes genomes that specify combinations of primitive update terms.<n>We find an evolved that outperforms Adam by 2.6% in aggregate fitness.
  arXiv  Detail & Related papers  (2025-12-05T01:30:07Z)
• Heart Disease Prediction: A Comparative Study of Optimisers Performance in Deep Neural Networks [0.0]
  We compare the performance of 10 different approaches in training a simple Multi-layer Perceptron model using a heart disease dataset from Kaggle.<n>We set up a consistent training paradigm and evaluate the metrics based on metrics such as convergence speed and stability.<n>Across all our metrics, we chose RMSProp to be the most effective for this heart disease prediction task because it offered a balanced performance across key metrics.
  arXiv  Detail & Related papers  (2025-09-10T11:15:44Z)
• Fantastic Pretraining Optimizers and Where to Find Them [59.56075036649332]
  AdamW has long been the dominant gradients in language model pretraining.<n>Speedup of matrix-based matrices is inversely proportional to model scale.
  arXiv  Detail & Related papers  (2025-09-02T07:43:22Z)
• A Stable Whitening Optimizer for Efficient Neural Network Training [101.89246340672246]
  Building on the Shampoo family of algorithms, we identify and alleviate three key issues, resulting in the proposed SPlus method.<n>First, we find that naive Shampoo is prone to divergence when matrix-inverses are cached for long periods.<n>Second, we adapt a shape-aware scaling to enable learning rate transfer across network width.<n>Third, we find that high learning rates result in large parameter noise, and propose a simple iterate-averaging scheme which unblocks faster learning.
  arXiv  Detail & Related papers  (2025-06-08T18:43:31Z)
• An Improved FOX Optimization Algorithm Using Adaptive Exploration and Exploitation for Global Optimization [3.88465206388773]
  This paper introduces an improved variation of the FOX optimization algorithm (FOX)<n>It incorporates a new adaptive method using a dynamically scaled step-size parameter to balance exploration and exploitation.<n>The experimental results showed that IFOX achieved a 40% improvement in overall performance metrics over the original FOX.
  arXiv  Detail & Related papers  (2025-04-13T13:50:18Z)
• EXAdam: The Power of Adaptive Cross-Moments [0.0]
  This paper introduces EXAdam, a novel optimization algorithm that builds upon the widely-used AdamAdam algorithm. EXAdam incorporates three key enhancements: (1) new debiasing terms for improved moment estimation, (2) a gradient-based acceleration mechanism, and (3) a dynamic step size formula. Empirical evaluations demonstrate EXAdam's superiority over Adam, achieving 48.07% faster convergence and yielding improvements of 4.6%, 4.13%, and 2.39% in training, validation, and testing accuracies.
  arXiv  Detail & Related papers  (2024-12-29T00:11:54Z)
• AutoScale: Automatic Prediction of Compute-optimal Data Composition for Training LLMs [61.13296177652599]
  This paper demonstrates that the optimal composition of training data from different domains is scale-dependent.<n>We introduce *AutoScale*, a novel, practical approach for optimizing data compositions at potentially large training data scales.<n>Our evaluation on GPT-2 Large and BERT pre-training demonstrates *AutoScale*'s effectiveness in improving training convergence and downstream performance.
  arXiv  Detail & Related papers  (2024-07-29T17:06:30Z)
• FOXANN: A Method for Boosting Neural Network Performance [4.8699089441419146]
  This paper presents FOXANN, a novel classification model that combines the recently developed Fox Artificial neural networks with ANN to solve ML problems. Fox replaces the backpropagation algorithm in ANN; optimize synaptic weights; and achieve high classification accuracy with a minimum loss, improved model generalization, and interpretability. The performance generalization of FOXANN is evaluated on three standard datasets: Iris Flower, Breast Cancer Wisconsin, and Wine.
  arXiv  Detail & Related papers  (2024-06-29T17:00:47Z)
• Iterative Reasoning Preference Optimization [84.15992372132507]
  We develop an iterative approach to optimize the preference between generated Chain-of-Thought (CoT) candidates. We show reasoning improves across repeated iterations of this scheme. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
  arXiv  Detail & Related papers  (2024-04-30T17:28:05Z)
• Adapprox: Adaptive Approximation in Adam Optimization via Randomized Low-Rank Matrices [24.319712013824876]
  Adapprox is a novel approach that employs randomized low-rank matrix approximation for a more accurate approximation of Adam's second moment. In GPT-2 training and downstream tasks, Adapprox surpasses AdamW by achieving 34.5% to 49.9% memory savings.
  arXiv  Detail & Related papers  (2024-03-22T05:23:31Z)
• Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning [126.84770886628833]
  Existing finetuning methods either tune all parameters of the pretrained model (full finetuning) or only tune the last linear layer (linear probing) We propose a new parameter-efficient finetuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance full finetuning.
  arXiv  Detail & Related papers  (2022-10-17T08:14:49Z)
• Distributed Evolution Strategies Using TPUs for Meta-Learning [0.0]
  We propose a distributed evolutionary meta-learning strategy using Processing Units (TPUs) that is highly parallel and scalable to arbitrarily long tasks with no increase in memory cost. Using a Prototypical Network trained with evolution strategies on the Omniglot dataset, we achieved an accuracy of 98.4% on a 5-shot classification problem. Our algorithm used as much as 40 times less memory than automatic differentiation to compute the gradient, with the resulting model achieving accuracy within 1.3% of a backpropagation-trained equivalent.
  arXiv  Detail & Related papers  (2022-01-01T02:14:02Z)
• Tom: Leveraging trend of the observed gradients for faster convergence [0.0]
  Tom is a novel variant of Adam that takes into account the trend observed for the gradients in the landscape in the loss traversed by the neural network. Tom outperforms Adagrad, Adadelta, RMSProp and Adam in terms of both accuracy and has a faster convergence.
  arXiv  Detail & Related papers  (2021-09-07T20:19:40Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)
• Inception Convolution with Efficient Dilation Search [121.41030859447487]
  Dilation convolution is a critical mutant of standard convolution neural network to control effective receptive fields and handle large scale variance of objects. We propose a new mutant of dilated convolution, namely inception (dilated) convolution where the convolutions have independent dilation among different axes, channels and layers. We explore a practical method for fitting the complex inception convolution to the data, a simple while effective dilation search algorithm(EDO) based on statistical optimization is developed.
  arXiv  Detail & Related papers  (2020-12-25T14:58:35Z)
• Modeling and Uncertainty Analysis of Groundwater Level Using Six Evolutionary Optimization Algorithms Hybridized with ANFIS, SVM, and ANN [0.0]
  Six meta-heuristic schemes are hybridized with artificial neural network (ANN), adaptive neuro-fuzzy interface system (ANFIS), and support vector machine (SVM) The performance of hybrid models with optimization algorithms was far better than that of classical ANN, ANFIS, and SVM models without hybridization.
  arXiv  Detail & Related papers  (2020-06-28T11:06:53Z)
• Smooth Adversarial Training [120.44430400607483]
  It is commonly believed that networks cannot be both accurate and robust. Here we present evidence to challenge these common beliefs by a careful study about adversarial training. We propose smooth adversarial training (SAT), in which we replace ReLU with its smooth approximations to strengthen adversarial training.
  arXiv  Detail & Related papers  (2020-06-25T16:34:39Z)
• ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning [91.13797346047984]
  We introduce ADAHESSIAN, a second order optimization algorithm which dynamically incorporates the curvature of the loss function via ADAptive estimates. We show that ADAHESSIAN achieves new state-of-the-art results by a large margin as compared to other adaptive optimization methods.
  arXiv  Detail & Related papers  (2020-06-01T05:00:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.