Benchmarking Optimizers for MLPs in Tabular Deep Learning

• URL: http://arxiv.org/abs/2604.15297v2
• Date: Fri, 17 Apr 2026 17:48:55 GMT
• Title: Benchmarking Optimizers for MLPs in Tabular Deep Learning
• Abstract summary: AdamW is the go-to training technique for training tabular deep learning models.<n>We benchmark 15 techniques on 17 datasets for training-based models in the standard supervised learning setting.<n>Our main finding is that the Muon consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers.
• Score: 26.245361419936135
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: MLP is a heavily used backbone in modern deep learning (DL) architectures for supervised learning on tabular data, and AdamW is the go-to optimizer used to train tabular DL models. Unlike architecture design, however, the choice of optimizer for tabular DL has not been examined systematically, despite new optimizers showing promise in other domains. To fill this gap, we benchmark 15 optimizers on 17 tabular datasets for training MLP-based models in the standard supervised learning setting under a shared experiment protocol. Our main finding is that the Muon optimizer consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers, if the associated training efficiency overhead is affordable. Additionally, we find exponential moving average of model weights to be a simple yet effective technique that improves AdamW on vanilla MLPs, though its effect is less consistent across model variants.

Related papers

• Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials [0.5714512533033075]
  We implement and compare a class of recently proposed matrix-structureds, including Muon, SOAP, and the hybrid SOAP-Muon, for training NequIP and Allegro MLIP models.<n>We find that theses can substantially outperform Adam in both convergence speed and final accuracy.
  arXiv  Detail & Related papers  (2026-07-02T17:57:31Z)
• iLTM: Integrated Large Tabular Model [41.81329403540607]
  iLTM is an integrated Large Tabular Model that unifies tree-derived embeddings, dimensionality-agnostic representations, a meta-trained hypernetwork, multilayer perceptrons, and retrieval within a single architecture.
  arXiv  Detail & Related papers  (2025-11-20T00:20:16Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Shadow-FT: Tuning Instruct Model via Training on Paired Base Model [67.20706292627106]
  Large language models (LLMs) consistently benefit from further fine-tuning on various tasks.<n>We propose a novel Shadow-FT framework to tune the Instruct models by leveraging the corresponding Base models.<n>Our proposed Shadow-FT introduces no additional parameters, is easy to implement, and significantly improves performance.
  arXiv  Detail & Related papers  (2025-05-19T05:16:21Z)
• (GG) MoE vs. MLP on Tabular Data [0.0]
  We introduce GG MoE, a mixture-of-experts (MoE) model with an embedding Gumbel-Softmax gating function.<n>We show that both MoE and GG MoE utilize significantly fewer parameters than neural networks.
  arXiv  Detail & Related papers  (2025-02-05T20:53:16Z)
• Adaptive Data Exploitation in Deep Reinforcement Learning [50.53705050673944]
  We introduce ADEPT, a powerful framework to enhance the **data efficiency** and **generalization** in deep reinforcement learning (RL)<n>Specifically, ADEPT adaptively manages the use of sampled data across different learning stages via multi-armed bandit (MAB) algorithms.<n>We test ADEPT on benchmarks including Procgen, MiniGrid, and PyBullet.
  arXiv  Detail & Related papers  (2025-01-22T04:01:17Z)
• TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling [28.37672139176765]
  New model TabM relies on ensembling, where one TabM efficiently imitates an ensemble ofs and produces multiple predictions per object.<n>In TabM, the underlying implicits are trained simultaneously, and (by default) share most of their parameters, which results in significantly better performance and efficiency.
  arXiv  Detail & Related papers  (2024-10-31T17:58:41Z)
• Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback [64.67540769692074]
  Large language models (LLMs) fine-tuned with alignment techniques, such as reinforcement learning from human feedback, have been instrumental in developing some of the most capable AI systems to date.<n>We introduce an approach called Margin Matching Preference Optimization (MMPO), which incorporates relative quality margins into optimization, leading to improved LLM policies and reward models.<n>Experiments with both human and AI feedback data demonstrate that MMPO consistently outperforms baseline methods, often by a substantial margin, on popular benchmarks including MT-bench and RewardBench.
  arXiv  Detail & Related papers  (2024-10-04T04:56:11Z)
• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback [110.16220825629749]
  Learning from preference feedback has emerged as an essential step for improving the generation quality and performance of modern language models. In this work, we identify four core aspects of preference-based learning: preference data, learning algorithm, reward model, and policy training prompts. Our findings indicate that all aspects are important for performance, with better preference data leading to the largest improvements.
  arXiv  Detail & Related papers  (2024-06-13T16:17:21Z)
• Active Preference Learning for Large Language Models [12.093302163058436]
  We develop an active learning strategy for DPO to make better use of preference labels. We propose a practical acquisition function for prompt/completion pairs based on the predictive entropy of the language model. We demonstrate how our approach improves both the rate of learning and final performance of fine-tuning on pairwise preference data.
  arXiv  Detail & Related papers  (2024-02-12T23:09:00Z)
• Tabular Data: Is Deep Learning all you need? [21.994741050350132]
  Tabular data represents one of the most prevalent data formats in applied machine learning.<n>Deep learning models have not been subjected to a comprehensive evaluation under conditions that allow for a fair comparison with existing classical approaches.<n>Our survey fills this gap by benchmarking seventeen state-of-the-art methods, spanning neural networks, classical ML and AutoML techniques.
  arXiv  Detail & Related papers  (2024-02-06T12:59:02Z)
• VeLO: Training Versatile Learned Optimizers by Scaling Up [67.90237498659397]
  We leverage the same scaling approach behind the success of deep learning to learn versatiles. We train an ingest for deep learning which is itself a small neural network that ingests and outputs parameter updates. We open source our learned, meta-training code, the associated train test data, and an extensive benchmark suite with baselines at velo-code.io.
  arXiv  Detail & Related papers  (2022-11-17T18:39:07Z)
• Revisiting Pretraining Objectives for Tabular Deep Learning [21.75490119265481]
  Pretraining is a workhorse of deep learning (DL) for vision and NLP. We show that using the object target labels during the pretraining stage is beneficial for the downstream performance.
  arXiv  Detail & Related papers  (2022-07-07T10:29:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.