Integer Natural Evolution Strategies

• URL: http://arxiv.org/abs/2608.23714v1
• Date: Mon, 24 Aug 2026 18:04:58 GMT
• Title: Integer Natural Evolution Strategies
• Abstract summary: We propose a fully $ell_1$-native step-size adaptation mechanism from first principles.<n>We show that the DG distribution belongs to the exponential family, and that its sufficient statistic $|z|$ yields a natural-gradient signal for adaptation.<n>This establishes that DG-based step-size adaptation arises directly from the statistical structure of the mutation distribution, rather than as a discrete analog of continuous ES mechanisms.
• Score: 3.6025363730694604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While contemporary Evolution Strategies handle integer optimization problems effectively, their adaptation mechanism is grounded in $\ell_2$-based Gaussian models, which are not native to the integer lattice. In contrast, the $\ell_1$-norm provides the natural measure of displacement on $\mathbb{Z}^n$, with the double geometric distribution as its canonical mutation operator. In this work, we derive a fully $\ell_1$-native step-size adaptation mechanism from first principles and propose an Integer Natural Evolution Strategy. We show that the DG distribution belongs to the exponential family, and that its sufficient statistic $|z|$ yields a natural-gradient signal for dispersion adaptation. By accumulating this signal via an evolution path, we obtain a fading-memory online estimator of the natural gradient, following Ollivier (2018). This establishes that DG-based step-size adaptation arises directly from the statistical structure of the mutation distribution, rather than as a discrete analog of continuous ES mechanisms. Empirical results on integer quadratic benchmarks show that \textsc{INES} learns meaningful coordinate-wise step-sizes and is competitive with integer-handling CMA-ES baselines. Its advantages are most visible in high-dimensional Ellipsoidal problems and in robust convergence at larger dimensions.

Related papers

• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation [56.361076943802594]
  CanonFlow achieves state-of-the-art performance on the challenging GEOM-DRUG dataset, and the advantage remains large in few-step generation.
  arXiv  Detail & Related papers  (2026-02-16T18:58:55Z)
• On the Computational Complexity of Performative Prediction [59.584063949469645]
  We show that computing an $$-performatively stable point is PPAD-complete.<n>One of our key technical contributions is to extend this PPAD-hardness result to general convex domains.
  arXiv  Detail & Related papers  (2026-01-28T02:26:35Z)
• Correlated Mutations for Integer Programming [0.0]
  This study seeks to establish the groundwork for Evolution Strategies (IESs)<n>IESs already excel in treating IP in practice, but accomplish it via discretization and by applying sophisticated patches to their continuous operators.<n>We focus on mutation distributions for integer decision variables.<n>We explore their theoretical properties, including entropy functions, and propose a procedure to generate scalable correlated mutation distributions.
  arXiv  Detail & Related papers  (2025-06-27T08:24:15Z)
• Mirror Descent and Novel Exponentiated Gradient Algorithms Using Trace-Form Entropies and Deformed Logarithms [17.422938130292827]
  This paper introduces a broad class of Mirror Descent (MD) and Generalized Exponentiated Gradient (GEG) algorithms.<n>Leveraging these generalized entropies yields MD & GEG algorithms with improved convergence behavior, robustness to vanishing and exploding gradients, and inherent adaptability to non-Euclidean geometries.
  arXiv  Detail & Related papers  (2025-03-11T10:50:07Z)
• Abnormal Mutations: Evolution Strategies Don't Require Gaussianity [3.9100702878800644]
  This paper empirically shows that a wide range of evolutionary strategies, from the (1+1)-ES to CMA-ES, show comparable optimization performance.<n>We observe that these results hold not only for the sphere model but also for a wider range of benchmark problems.
  arXiv  Detail & Related papers  (2025-02-05T13:17:07Z)
• Computational-Statistical Gaps in Gaussian Single-Index Models [77.1473134227844]
  Single-Index Models are high-dimensional regression problems with planted structure. We show that computationally efficient algorithms, both within the Statistical Query (SQ) and the Low-Degree Polynomial (LDP) framework, necessarily require $Omega(dkstar/2)$ samples.
  arXiv  Detail & Related papers  (2024-03-08T18:50:19Z)
• Implicit Bias and Fast Convergence Rates for Self-attention [26.766649949420746]
  We study the fundamental optimization principles of self-attention, the defining mechanism of transformers.<n>We analyze the implicit bias of gradient-baseds in a self-attention layer with a decoder in a linear classification.
  arXiv  Detail & Related papers  (2024-02-08T15:15:09Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer [45.47667026025716]
  We propose a novel, robust and accelerated iteration that relies on two key elements. The convergence and stability of the obtained method, referred to as NAG-GS, are first studied extensively. We show that NAG-arity is competitive with state-the-art methods such as momentum SGD with weight decay and AdamW for the training of machine learning models.
  arXiv  Detail & Related papers  (2022-09-29T16:54:53Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-$\ell_1$-Norm Interpolated Classifiers [3.167685495996986]
  This paper establishes a precise high-dimensional theory for boosting on separable data. Under a class of statistical models, we provide an exact analysis of the universality error of boosting. We also explicitly pin down the relation between the boosting test error and the optimal Bayes error.
  arXiv  Detail & Related papers  (2020-02-05T00:24:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.