Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection

• URL: http://arxiv.org/abs/2608.18810v1
• Date: Wed, 19 Aug 2026 11:08:42 GMT
• Title: Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection
• Abstract summary: Repeated Resampling (ROR) searches during one evolving run.<n>One-epoch ROR uses 24 to 35% of the aggregate training needed to identify the best fixed variants exhaustively.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An optimizer is usually chosen before training a deep neural network and then kept fixed. Treating optimizer choice as a hyperparameter could boost performance, but it requires several complete training runs and discards all but the winner. Repeated Optimizer Resampling (ROR) instead searches during one evolving run. Every $b$ epochs, each candidate optimizer scouts from the current model weights for $s$ epochs. The best scout continues for the remaining $b-s$ epochs, and that completed segment becomes the new incumbent if it improves the validation objective. This design allows the preferred optimizer to change as training progresses. We compare two variants of ROR on MNIST, Fashion-MNIST, and two motor insurance claim-count models. Nine fixed optimizers and both ROR variants are evaluated with the same ten seeds. One-epoch ROR uses 24\% to 35\% of the aggregate training needed to identify the best fixed optimizer exhaustively and remains close to that optimizer on all four tasks. These results support short scouting as a practical way to search over optimizers without completing every candidate run.

Related papers

• Reinforcement learning to choose optimizers [0.0]
  "Reinforcement Learning to Chooses" employs an algorithm choice as a sequential decision-making problem.<n>The portfolio includes gradient-based and derivative-frees, and each switch passes on the current best solution and a representative step size.<n>On unseen problems, the learned policy outperforms every portfolio at all but the smallest budgets, and it remains robust under distribution shift.
  arXiv  Detail & Related papers  (2026-09-01T19:35:31Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• Cautious Optimizers: Improving Training with One Line of Code [8.393403749426097]
  We rename AdamW as cautious, e.g. C-AdamW and C-Lion.<n>Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis.<n>Among them, we pick the simplest one for empirical experiments, showing not only speed-up on Llama and MAE pretraining up to $1.47$ times, but also better results in post-training tasks.
  arXiv  Detail & Related papers  (2024-11-25T04:36:01Z)
• Iterative Reasoning Preference Optimization [84.15992372132507]
  We develop an iterative approach to optimize the preference between generated Chain-of-Thought (CoT) candidates. We show reasoning improves across repeated iterations of this scheme. For example, we see a large improvement from 55.6% to 81.6% on GSM8K and an accuracy of 88.7% with majority voting out of 32 samples.
  arXiv  Detail & Related papers  (2024-04-30T17:28:05Z)
• Should I try multiple optimizers when fine-tuning pre-trained Transformers for NLP tasks? Should I tune their hyperparameters? [14.349943044268471]
  Gradient Descent (SGD) is employed to select neural models for training. tuning just the learning rate is in most cases as good as tuning all the hyperparameters. We recommend picking any of the best-behaved adaptiveBounds (e.g., Adam) and recommending its learning rate.
  arXiv  Detail & Related papers  (2024-02-10T13:26:14Z)
• MADA: Meta-Adaptive Optimizers through hyper-gradient Descent [73.1383658672682]
  We introduce Meta-Adaptives (MADA), a unified framework that can generalize several known convergences and dynamically learn the most suitable one during training. We empirically compare MADA to other populars on vision and language tasks, and find that MADA consistently outperforms Adam and other populars. We also propose AVGrad, a modification of AMS that replaces the maximum operator with averaging, which is more suitable for hyper-gradient optimization.
  arXiv  Detail & Related papers  (2024-01-17T00:16:46Z)
• CoRe Optimizer: An All-in-One Solution for Machine Learning [0.0]
  Continuously resilient convergence (CoRe) shown superior performance compared to other state-of-the-art first-order gradient-based convergence algorithms. CoRe yields best or competitive performance in every investigated application.
  arXiv  Detail & Related papers  (2023-07-28T16:48:42Z)
• VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
  We leverage the same scaling approach behind the success of deep learning to learn versatiles. We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates. We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
  arXiv  Detail & Related papers  (2022-11-17T18:39:07Z)
• VSAC: Efficient and Accurate Estimator for H and F [68.65610177368617]
  VSAC is a RANSAC-type robust estimator with a number of novelties. It is significantly faster than all its predecessors and runs on average in 1-2 ms, on a CPU. It is two orders of magnitude faster and yet as precise as MAGSAC++, the currently most accurate estimator of two-view geometry.
  arXiv  Detail & Related papers  (2021-06-18T17:04:57Z)
• Training Learned Optimizers with Randomly Initialized Learned Optimizers [49.67678615506608]
  We show that a population of randomly learneds can be used to train themselves from scratch in an online fashion. A form of population based training is used to orchestrate this self-training. We believe feedback loops of this type will be important and powerful in the future of machine learning.
  arXiv  Detail & Related papers  (2021-01-14T19:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.