Fast Moving Natural Evolution Strategy for High-Dimensional Problems

• URL: http://arxiv.org/abs/2201.11422v1
• Date: Thu, 27 Jan 2022 10:18:11 GMT
• Title: Fast Moving Natural Evolution Strategy for High-Dimensional Problems
• Authors: Masahiro Nomura, Isao Ono
• Abstract summary: We propose a new variant of natural evolution strategies (NES) for high-dimensional black-box optimization problems. The proposed method, CR-FM-NES, extends a recently proposed state-of-the-art NES, Fast Moving Natural Evolution Strategy (FM-NES) in order to be applicable in high-dimensional problems.
• Score: 2.512827436728378
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we propose a new variant of natural evolution strategies (NES) for high-dimensional black-box optimization problems. The proposed method, CR-FM-NES, extends a recently proposed state-of-the-art NES, Fast Moving Natural Evolution Strategy (FM-NES), in order to be applicable in high-dimensional problems. CR-FM-NES builds on an idea using a restricted representation of a covariance matrix instead of using a full covariance matrix, while inheriting an efficiency of FM-NES. The restricted representation of the covariance matrix enables CR-FM-NES to update parameters of a multivariate normal distribution in linear time and space complexity, which can be applied to high-dimensional problems. Our experimental results reveal that CR-FM-NES does not lose the efficiency of FM-NES, and on the contrary, CR-FM-NES has achieved significant speedup compared to FM-NES on some benchmark problems. Furthermore, our numerical experiments using 200, 600, and 1000-dimensional benchmark problems demonstrate that CR-FM-NES is effective over scalable baseline methods, VD-CMA and Sep-CMA.

Related papers

• Adversarial Schrödinger Bridge Matching [66.39774923893103]
  Iterative Markovian Fitting (IMF) procedure alternates between Markovian and reciprocal projections of continuous-time processes. We propose a novel Discrete-time IMF (D-IMF) procedure in which learning of processes is replaced by learning just a few transition probabilities in discrete time. We show that our D-IMF procedure can provide the same quality of unpaired domain translation as the IMF, using only several generation steps instead of hundreds.
  arXiv  Detail & Related papers  (2024-05-23T11:29:33Z)
• Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models [73.48675708831328]
  We propose a novel parameter and computation efficient tuning method for Multi-modal Large Language Models (MLLMs) The Efficient Attention Skipping (EAS) method evaluates the attention redundancy and skips the less important MHAs to speed up inference. The experiments show that EAS not only retains high performance and parameter efficiency, but also greatly speeds up inference speed.
  arXiv  Detail & Related papers  (2024-03-22T14:20:34Z)
• Transforming Image Super-Resolution: A ConvFormer-based Efficient Approach [58.57026686186709]
  We introduce the Convolutional Transformer layer (ConvFormer) and propose a ConvFormer-based Super-Resolution network (CFSR) CFSR inherits the advantages of both convolution-based and transformer-based approaches. Experiments demonstrate that CFSR strikes an optimal balance between computational cost and performance.
  arXiv  Detail & Related papers  (2024-01-11T03:08:00Z)
• Natural Evolution Strategy for Mixed-Integer Black-Box Optimization [0.0]
  CMA-ES w. Margin reportedly showed good performance on MI-BBO benchmark problems. DX-NES-ICI was up to 3.7 times better than CMA-ES w. Margin in terms of a rate of finding the optimal solutions.
  arXiv  Detail & Related papers  (2023-04-21T03:34:22Z)
• Online Probabilistic Model Identification using Adaptive Recursive MCMC [8.465242072268019]
  We suggest the Adaptive Recursive Markov Chain Monte Carlo (ARMCMC) method. It eliminates the shortcomings of conventional online techniques while computing the entire probability density function of model parameters. We demonstrate our approach using parameter estimation in a soft bending actuator and the Hunt-Crossley dynamic model.
  arXiv  Detail & Related papers  (2022-10-23T02:06:48Z)
• Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning [63.83425382922157]
  Intelligent reflecting surface (IRS) is envisioned to be widely applied in future wireless networks. In this paper, we investigate a multi-user communication system assisted by cooperative IRS devices with the capability of energy harvesting.
  arXiv  Detail & Related papers  (2022-03-26T20:37:14Z)
• FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalization [73.41395947275473]
  We propose a novel frequency-aware architecture, in which the domain-specific features are filtered out in the transformed frequency domain. Experiments on three benchmarks demonstrate significant performance, outperforming the state-of-the-art methods by a margin of 3%, 4% and 9%, respectively.
  arXiv  Detail & Related papers  (2022-03-24T07:26:29Z)
• NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM [16.464030458567187]
  We introduce a new learning framework, called NxMTransformer, to induce NxM semi-structured sparsity on pretrained language models. We propose to formulate the NxM sparsity as a constrained optimization problem and use Alternating Direction Method of Multipliers (ADMM) to optimize the downstream tasks. Our proposed method is able to achieve 1.7 points higher accuracy in GLUE score than current practices.
  arXiv  Detail & Related papers  (2021-10-28T17:43:06Z)
• Natural Evolution Strategy for Unconstrained and Implicitly Constrained Problems with Ridge Structure [2.512827436728378]
  We propose a new natural evolution strategy for unconstrained black-box function optimization (BBFO) problems and implicitly constrained BBFO problems. FM-NES shows better performance than DX-NES-IC on problems with ridge structure and almost the same performance as DX-NES-IC on the others.
  arXiv  Detail & Related papers  (2021-08-21T08:01:56Z)
• Mixed Variable Bayesian Optimization with Frequency Modulated Kernels [96.78099706164747]
  We propose the frequency modulated (FM) kernel flexibly modeling dependencies among different types of variables. BO-FM outperforms competitors including Regularized evolution(RE) and BOHB.
  arXiv  Detail & Related papers  (2021-02-25T11:28:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.