Optimizing Training Trajectories in Variational Autoencoders via Latent
Bayesian Optimization Approach
- URL: http://arxiv.org/abs/2207.00128v1
- Date: Thu, 30 Jun 2022 23:41:47 GMT
- Title: Optimizing Training Trajectories in Variational Autoencoders via Latent
Bayesian Optimization Approach
- Authors: Arpan Biswas, Rama Vasudevan, Maxim Ziatdinov, Sergei V. Kalinin
- Abstract summary: Unsupervised and semi-supervised ML methods have become widely adopted across multiple areas of physics, chemistry, and materials sciences.
We propose a latent Bayesian optimization (zBO) approach for the hyper parameter trajectory optimization for the unsupervised and semi-supervised ML.
We demonstrate an application of this method for finding joint discrete and continuous rotationally invariant representations for MNIST and experimental data of a plasmonic nanoparticles material system.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Unsupervised and semi-supervised ML methods such as variational autoencoders
(VAE) have become widely adopted across multiple areas of physics, chemistry,
and materials sciences due to their capability in disentangling representations
and ability to find latent manifolds for classification and regression of
complex experimental data. Like other ML problems, VAEs require hyperparameter
tuning, e.g., balancing the Kullback Leibler (KL) and reconstruction terms.
However, the training process and resulting manifold topology and connectivity
depend not only on hyperparameters, but also their evolution during training.
Because of the inefficiency of exhaustive search in a high-dimensional
hyperparameter space for the expensive to train models, here we explored a
latent Bayesian optimization (zBO) approach for the hyperparameter trajectory
optimization for the unsupervised and semi-supervised ML and demonstrate for
joint-VAE with rotational invariances. We demonstrate an application of this
method for finding joint discrete and continuous rotationally invariant
representations for MNIST and experimental data of a plasmonic nanoparticles
material system. The performance of the proposed approach has been discussed
extensively, where it allows for any high dimensional hyperparameter tuning or
trajectory optimization of other ML models.
Related papers
- Spectrum-Aware Parameter Efficient Fine-Tuning for Diffusion Models [73.88009808326387]
We propose a novel spectrum-aware adaptation framework for generative models.
Our method adjusts both singular values and their basis vectors of pretrained weights.
We introduce Spectral Ortho Decomposition Adaptation (SODA), which balances computational efficiency and representation capacity.
arXiv Detail & Related papers (2024-05-31T17:43:35Z) - Dual-Space Optimization: Improved Molecule Sequence Design by Latent
Prompt Transformer [78.47949363282868]
We propose the Dual-Space Optimization (DSO) method that integrates latent space sampling and data space selection to solve this problem.
DSO iteratively updates a latent space generative model and a synthetic dataset in an optimization process that shifts the generative model and the synthetic data towards regions of desired property values.
Our experiments demonstrate effectiveness of the proposed method, which sets new performance benchmarks across single-objective, multi-objective and constrained molecule design tasks.
arXiv Detail & Related papers (2024-02-27T03:33:23Z) - Online Variational Sequential Monte Carlo [49.97673761305336]
We build upon the variational sequential Monte Carlo (VSMC) method, which provides computationally efficient and accurate model parameter estimation and Bayesian latent-state inference.
Online VSMC is capable of performing efficiently, entirely on-the-fly, both parameter estimation and particle proposal adaptation.
arXiv Detail & Related papers (2023-12-19T21:45:38Z) - Subsurface Characterization using Ensemble-based Approaches with Deep
Generative Models [2.184775414778289]
Inverse modeling is limited for ill-posed, high-dimensional applications due to computational costs and poor prediction accuracy with sparse datasets.
We combine Wasserstein Geneversarative Adrial Network with Gradient Penalty (WGAN-GP) and Ensemble Smoother with Multiple Data Assimilation (ES-MDA)
WGAN-GP is trained to generate high-dimensional K fields from a low-dimensional latent space and ES-MDA updates the latent variables by assimilating available measurements.
arXiv Detail & Related papers (2023-10-02T01:27:10Z) - Active-Learning-Driven Surrogate Modeling for Efficient Simulation of
Parametric Nonlinear Systems [0.0]
In absence of governing equations, we need to construct the parametric reduced-order surrogate model in a non-intrusive fashion.
Our work provides a non-intrusive optimality criterion to efficiently populate the parameter snapshots.
We propose an active-learning-driven surrogate model using kernel-based shallow neural networks.
arXiv Detail & Related papers (2023-06-09T18:01:14Z) - VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables.
The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning.
We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
arXiv Detail & Related papers (2023-04-03T13:13:19Z) - An Adaptive Alternating-direction-method-based Nonnegative Latent Factor
Model [2.857044909410376]
An alternating-direction-method-based nonnegative latent factor model can perform efficient representation learning to a high-dimensional and incomplete (HDI) matrix.
This paper proposes an Adaptive Alternating-direction-method-based Nonnegative Latent Factor model, whose hyper- parameter adaptation is implemented following the principle of particle swarm optimization.
Empirical studies on nonnegative HDI matrices generated by industrial applications indicate that A2NLF outperforms several state-of-the-art models in terms of computational and storage efficiency, as well as maintains highly competitive estimation accuracy for an HDI matrix's missing data
arXiv Detail & Related papers (2022-04-11T03:04:26Z) - Equivariant vector field network for many-body system modeling [65.22203086172019]
Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers.
We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
arXiv Detail & Related papers (2021-10-26T14:26:25Z) - Spectral Tensor Train Parameterization of Deep Learning Layers [136.4761580842396]
We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context.
We show the effects of neural network compression in the classification setting and both compression and improved stability training in the generative adversarial training setting.
arXiv Detail & Related papers (2021-03-07T00:15:44Z) - Bayesian multiscale deep generative model for the solution of
high-dimensional inverse problems [0.0]
A novel multiscale Bayesian inference approach is introduced based on deep probabilistic generative models.
The method allows high-dimensional parameter estimation while exhibiting stability, efficiency and accuracy.
arXiv Detail & Related papers (2021-02-04T11:47:21Z) - VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization [4.237343083490243]
During the training phase of machine learning (ML) models, it is usually necessary to configure several hyper parameters.
We present VisEvol, a visual analytics tool that supports interactive exploration of hyper parameters and intervention in this evolutionary procedure.
The utility and applicability of VisEvol are demonstrated with two use cases and interviews with ML experts who evaluated the effectiveness of the tool.
arXiv Detail & Related papers (2020-12-02T13:43:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.