Related papers: Continuous Function Structured in Multilayer Perceptron for Global Optimization

Continuous Function Structured in Multilayer Perceptron for Global Optimization

URL: http://arxiv.org/abs/2303.04623v1
Date: Tue, 7 Mar 2023 14:50:50 GMT
Title: Continuous Function Structured in Multilayer Perceptron for Global Optimization
Authors: Heeyuen Koh
Abstract summary: gradient information of multilayer perceptron with a linear neuron is modified with functional derivative for benchmarking global minimum search problems. We show that the landscape of the gradient derived from given continuous function using functional derivative can be a form with ax+b neurons.
Score: 0.0
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: The gradient information of multilayer perceptron with a linear neuron is modified with functional derivative for the global minimum search benchmarking problems. From this approach, we show that the landscape of the gradient derived from given continuous function using functional derivative can be the MLP-like form with ax+b neurons. In this extent, the suggested algorithm improves the availability of the optimization process to deal all the parameters in the problem set simultaneously. The functionality of this method could be improved through intentionally designed convex function with Kullack-Liebler divergence applied to cost value as well.

Related papers

Global Optimization of Gaussian Process Acquisition Functions Using a Piecewise-Linear Kernel Approximation [2.3342885570554652]
We introduce a piecewise approximation for process kernels and a corresponding MIQP representation for acquisition functions. We empirically demonstrate the framework on synthetic functions, constrained benchmarks, and hyper tuning tasks.
arXiv Detail & Related papers (2024-10-22T10:56:52Z)
A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
arXiv Detail & Related papers (2024-04-18T16:46:08Z)
On Learning Gaussian Multi-index Models with Gradient Flow [57.170617397894404]
We study gradient flow on the multi-index regression problem for high-dimensional Gaussian data. We consider a two-timescale algorithm, whereby the low-dimensional link function is learnt with a non-parametric model infinitely faster than the subspace parametrizing the low-rank projection.
arXiv Detail & Related papers (2023-10-30T17:55:28Z)
Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels [78.6096486885658]
We introduce lower bounds to the linearized Laplace approximation of the marginal likelihood. These bounds are amenable togradient-based optimization and allow to trade off estimation accuracy against computational complexity.
arXiv Detail & Related papers (2023-06-06T19:02:57Z)
Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
arXiv Detail & Related papers (2023-04-17T14:23:43Z)
$\mathcal{C}^k$-continuous Spline Approximation with TensorFlow Gradient Descent Optimizers [2.0305676256390934]
We present an "out-of-the-box" application of Machine Learning (ML)s for an industrial optimization problem. We introduce a piecewise model (spline) for fitting of $mathcalCk$-continuos functions, which can be deployed in a cam approximation setting. We then use the gradient descent optimization provided by the machine learning framework to optimize the model parameters with respect to approximation quality and $mathcalCk$-continuity.
arXiv Detail & Related papers (2023-03-22T10:52:21Z)
On Solution Functions of Optimization: Universal Approximation and Covering Number Bounds [6.3291148076593355]
We study the expressability of convex optimization functions solution functions of linear targetibility(1) (LP) and approximable approximation power of QP. Our results provide the emph we model for rigorous analysis of the properties of some restricted programming and implications for algorithmic design and performance guarantees.
arXiv Detail & Related papers (2022-12-02T17:16:04Z)
Implicit differentiation for fast hyperparameter selection in non-smooth convex learning [87.60600646105696]
We study first-order methods when the inner optimization problem is convex but non-smooth. We show that the forward-mode differentiation of proximal gradient descent and proximal coordinate descent yield sequences of Jacobians converging toward the exact Jacobian.
arXiv Detail & Related papers (2021-05-04T17:31:28Z)
CDiNN -Convex Difference Neural Networks [0.8122270502556374]
Neural networks with ReLU activation function have been shown to be universal function approximators learn function mapping as non-smooth functions. New neural network architecture called ICNNs learn the output as a convex input.
arXiv Detail & Related papers (2021-03-31T17:31:16Z)
SLEIPNIR: Deterministic and Provably Accurate Feature Expansion for Gaussian Process Regression with Derivatives [86.01677297601624]
We propose a novel approach for scaling GP regression with derivatives based on quadrature Fourier features. We prove deterministic, non-asymptotic and exponentially fast decaying error bounds which apply for both the approximated kernel as well as the approximated posterior.
arXiv Detail & Related papers (2020-03-05T14:33:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.