Mixed Variable Bayesian Optimization with Frequency Modulated Kernels
- URL: http://arxiv.org/abs/2102.12792v1
- Date: Thu, 25 Feb 2021 11:28:46 GMT
- Title: Mixed Variable Bayesian Optimization with Frequency Modulated Kernels
- Authors: Changyong Oh, Efstratios Gavves, Max Welling
- Abstract summary: We propose the frequency modulated (FM) kernel flexibly modeling dependencies among different types of variables.
BO-FM outperforms competitors including Regularized evolution(RE) and BOHB.
- Score: 96.78099706164747
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The sample efficiency of Bayesian optimization(BO) is often boosted by
Gaussian Process(GP) surrogate models. However, on mixed variable spaces,
surrogate models other than GPs are prevalent, mainly due to the lack of
kernels which can model complex dependencies across different types of
variables. In this paper, we propose the frequency modulated (FM) kernel
flexibly modeling dependencies among different types of variables, so that BO
can enjoy the further improved sample efficiency. The FM kernel uses distances
on continuous variables to modulate the graph Fourier spectrum derived from
discrete variables. However, the frequency modulation does not always define a
kernel with the similarity measure behavior which returns higher values for
pairs of more similar points. Therefore, we specify and prove conditions for FM
kernels to be positive definite and to exhibit the similarity measure behavior.
In experiments, we demonstrate the improved sample efficiency of GP BO using FM
kernels (BO-FM).On synthetic problems and hyperparameter optimization problems,
BO-FM outperforms competitors consistently. Also, the importance of the
frequency modulation principle is empirically demonstrated on the same
problems. On joint optimization of neural architectures and SGD
hyperparameters, BO-FM outperforms competitors including Regularized
evolution(RE) and BOHB. Remarkably, BO-FM performs better even than RE and BOHB
using three times as many evaluations.
Related papers
- Local Flow Matching Generative Models [19.859984725284896]
Flow Matching (FM) is a simulation-free method for learning a continuous and invertible flow to interpolate between two distributions.
We introduce Local Flow Matching (LFM), which learns a sequence of FM sub-models and each matches a diffusion process up to the time of the step size in the data-to-noise direction.
In experiments, we demonstrate the improved training efficiency and competitive generative performance of LFM compared to FM.
arXiv Detail & Related papers (2024-10-03T14:53:10Z) - Consistency Flow Matching: Defining Straight Flows with Velocity Consistency [97.28511135503176]
We introduce Consistency Flow Matching (Consistency-FM), a novel FM method that explicitly enforces self-consistency in the velocity field.
Preliminary experiments demonstrate that our Consistency-FM significantly improves training efficiency by converging 4.4x faster than consistency models.
arXiv Detail & Related papers (2024-07-02T16:15:37Z) - Variance-Reducing Couplings for Random Features [57.73648780299374]
Random features (RFs) are a popular technique to scale up kernel methods in machine learning.
We find couplings to improve RFs defined on both Euclidean and discrete input spaces.
We reach surprising conclusions about the benefits and limitations of variance reduction as a paradigm.
arXiv Detail & Related papers (2024-05-26T12:25:09Z) - Switched Flow Matching: Eliminating Singularities via Switching ODEs [12.273757042838675]
Continuous-time generative models, such as Flow Matching (FM), construct probability paths to transport between one distribution and another.
During inference, however, the learned model often requires multiple neural network evaluations to accurately integrate the flow.
We propose Switched FM (SFM), that eliminates singularities via switching ODEs, as opposed to using a uniform ODE in FM.
arXiv Detail & Related papers (2024-05-19T16:21:04Z) - Improving and generalizing flow-based generative models with minibatch
optimal transport [90.01613198337833]
We introduce the generalized conditional flow matching (CFM) technique for continuous normalizing flows (CNFs)
CFM features a stable regression objective like that used to train the flow in diffusion models but enjoys the efficient inference of deterministic flow models.
A variant of our objective is optimal transport CFM (OT-CFM), which creates simpler flows that are more stable to train and lead to faster inference.
arXiv Detail & Related papers (2023-02-01T14:47:17Z) - Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time.
This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
arXiv Detail & Related papers (2022-11-26T01:56:05Z) - FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalization [73.41395947275473]
We propose a novel frequency-aware architecture, in which the domain-specific features are filtered out in the transformed frequency domain.
Experiments on three benchmarks demonstrate significant performance, outperforming the state-of-the-art methods by a margin of 3%, 4% and 9%, respectively.
arXiv Detail & Related papers (2022-03-24T07:26:29Z) - Modelling Non-Smooth Signals with Complex Spectral Structure [26.749261270690432]
We redesign the GPCM model to induce a richer distribution over the spectrum with relaxed assumptions about smoothness.
We also propose a more effective variational inference scheme, going beyond the mean-field assumption.
The proposed variations of the GPCM are validated in experiments on synthetic and real-world data, showing promising results.
arXiv Detail & Related papers (2022-03-14T11:02:38Z) - Approximate Inference for Spectral Mixture Kernel [25.087829816206813]
We propose an approximate Bayesian inference for the spectral mixture kernel.
We optimize the variational parameters by applying a sampling-based variational inference to the derived evidence lower bound (ELBO) estimator.
The proposed inference combined with two strategies accelerates the convergence of the parameters and leads to better optimal parameters.
arXiv Detail & Related papers (2020-06-12T09:39:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.