Modeling Design and Control Problems Involving Neural Network Surrogates
- URL: http://arxiv.org/abs/2111.10489v1
- Date: Sat, 20 Nov 2021 01:09:15 GMT
- Title: Modeling Design and Control Problems Involving Neural Network Surrogates
- Authors: Dominic Yang, Prasanna Balaprakash, Sven Leyffer
- Abstract summary: We consider nonlinear optimization problems that involve surrogate models represented by neural networks.
We show how to directly embed neural network evaluation into optimization models, highlight a difficulty with this approach that can prevent convergence.
We present two alternative formulations of these problems in the specific case of feedforward neural networks with ReLU activation.
- Score: 1.1602089225841632
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We consider nonlinear optimization problems that involve surrogate models
represented by neural networks. We demonstrate first how to directly embed
neural network evaluation into optimization models, highlight a difficulty with
this approach that can prevent convergence, and then characterize stationarity
of such models. We then present two alternative formulations of these problems
in the specific case of feedforward neural networks with ReLU activation: as a
mixed-integer optimization problem and as a mathematical program with
complementarity constraints. For the latter formulation we prove that
stationarity at a point for this problem corresponds to stationarity of the
embedded formulation. Each of these formulations may be solved with
state-of-the-art optimization methods, and we show how to obtain good initial
feasible solutions for these methods. We compare our formulations on three
practical applications arising in the design and control of combustion engines,
in the generation of adversarial attacks on classifier networks, and in the
determination of optimal flows in an oil well network.
Related papers
- Diffusion Models as Network Optimizers: Explorations and Analysis [71.69869025878856]
generative diffusion models (GDMs) have emerged as a promising new approach to network optimization.
In this study, we first explore the intrinsic characteristics of generative models.
We provide a concise theoretical and intuitive demonstration of the advantages of generative models over discriminative network optimization.
arXiv Detail & Related papers (2024-11-01T09:05:47Z) - A Primal-dual algorithm for image reconstruction with ICNNs [3.4797100095791706]
We address the optimization problem in a data-driven variational framework, where the regularizer is parameterized by an input- neural network (ICNN)
While gradient-based methods are commonly used to solve such problems, they struggle to effectively handle nonsmoothness.
We show that a proposed approach outperforms subgradient methods in terms of both speed and stability.
arXiv Detail & Related papers (2024-10-16T10:36:29Z) - Rational-WENO: A lightweight, physically-consistent three-point weighted essentially non-oscillatory scheme [14.120671138290104]
We employ a rational neural network to accurately estimate the local smoothness of the solution.
This approach achieves a granular reconstruction with significantly reduced dissipation.
We demonstrate the effectiveness of our approach on several one-, two-, and three-dimensional fluid flow problems.
arXiv Detail & Related papers (2024-09-13T22:11:03Z) - Reducing the Need for Backpropagation and Discovering Better Optima With
Explicit Optimizations of Neural Networks [4.807347156077897]
We propose a computationally efficient alternative for optimizing neural networks.
We derive an explicit solution to a simple feed-forward language model.
We show that explicit solutions perform near-optimality in experiments.
arXiv Detail & Related papers (2023-11-13T17:38:07Z) - An Optimization-based Deep Equilibrium Model for Hyperspectral Image
Deconvolution with Convergence Guarantees [71.57324258813675]
We propose a novel methodology for addressing the hyperspectral image deconvolution problem.
A new optimization problem is formulated, leveraging a learnable regularizer in the form of a neural network.
The derived iterative solver is then expressed as a fixed-point calculation problem within the Deep Equilibrium framework.
arXiv Detail & Related papers (2023-06-10T08:25:16Z) - A Stable and Scalable Method for Solving Initial Value PDEs with Neural
Networks [52.5899851000193]
We develop an ODE based IVP solver which prevents the network from getting ill-conditioned and runs in time linear in the number of parameters.
We show that current methods based on this approach suffer from two key issues.
First, following the ODE produces an uncontrolled growth in the conditioning of the problem, ultimately leading to unacceptably large numerical errors.
arXiv Detail & Related papers (2023-04-28T17:28:18Z) - Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks.
One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver.
This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
arXiv Detail & Related papers (2023-01-28T01:50:42Z) - Acceleration techniques for optimization over trained neural network
ensembles [1.0323063834827415]
We study optimization problems where the objective function is modeled through feedforward neural networks with rectified linear unit activation.
We present a mixed-integer linear program based on existing popular big-$M$ formulations for optimizing over a single neural network.
arXiv Detail & Related papers (2021-12-13T20:50:54Z) - Offline Model-Based Optimization via Normalized Maximum Likelihood
Estimation [101.22379613810881]
We consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points.
This problem setting emerges in many domains where function evaluation is a complex and expensive process.
We propose a tractable approximation that allows us to scale our method to high-capacity neural network models.
arXiv Detail & Related papers (2021-02-16T06:04:27Z) - Automatically Learning Compact Quality-aware Surrogates for Optimization
Problems [55.94450542785096]
Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values.
Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making.
We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
arXiv Detail & Related papers (2020-06-18T19:11:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.