Adaptive Projected Residual Networks for Learning Parametric Maps from
Sparse Data
- URL: http://arxiv.org/abs/2112.07096v1
- Date: Tue, 14 Dec 2021 01:29:19 GMT
- Title: Adaptive Projected Residual Networks for Learning Parametric Maps from
Sparse Data
- Authors: Thomas O'Leary-Roseberry, Xiaosong Du, Anirban Chaudhuri, Joaquim R.
R. A. Martins, Karen Willcox and Omar Ghattas
- Abstract summary: We present a parsimonious surrogate framework for learning high dimensional parametric maps from limited training data.
These applications include such "outer-loop" problems as Bayesian inverse problems, optimal experimental design, and optimal design and control under uncertainty.
- Score: 5.920947681019466
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a parsimonious surrogate framework for learning high dimensional
parametric maps from limited training data. The need for parametric surrogates
arises in many applications that require repeated queries of complex
computational models. These applications include such "outer-loop" problems as
Bayesian inverse problems, optimal experimental design, and optimal design and
control under uncertainty, as well as real time inference and control problems.
Many high dimensional parametric mappings admit low dimensional structure,
which can be exploited by mapping-informed reduced bases of the inputs and
outputs. Exploiting this property, we develop a framework for learning low
dimensional approximations of such maps by adaptively constructing ResNet
approximations between reduced bases of their inputs and output. Motivated by
recent approximation theory for ResNets as discretizations of control flows, we
prove a universal approximation property of our proposed adaptive projected
ResNet framework, which motivates a related iterative algorithm for the ResNet
construction. This strategy represents a confluence of the approximation theory
and the algorithm since both make use of sequentially minimizing flows. In
numerical examples we show that these parsimonious, mapping-informed
architectures are able to achieve remarkably high accuracy given few training
data, making them a desirable surrogate strategy to be implemented for minimal
computational investment in training data generation.
Related papers
- Symplectic Autoencoders for Model Reduction of Hamiltonian Systems [0.0]
It is crucial to preserve the symplectic structure associated with the system in order to ensure long-term numerical stability.
We propose a new neural network architecture in the spirit of autoencoders, which are established tools for dimension reduction.
In order to train the network, a non-standard gradient descent approach is applied.
arXiv Detail & Related papers (2023-12-15T18:20:25Z) - Adaptive operator learning for infinite-dimensional Bayesian inverse
problems [8.672948020721945]
We develop an adaptive operator learning framework that can reduce modeling error gradually by forcing the surrogate to be accurate in local areas.
We present a rigorous convergence guarantee in the linear case using the UKI framework.
The numerical results show that our method can significantly reduce computational costs while maintaining inversion accuracy.
arXiv Detail & Related papers (2023-10-27T01:50:33Z) - Towards a Better Theoretical Understanding of Independent Subnetwork Training [56.24689348875711]
We take a closer theoretical look at Independent Subnetwork Training (IST)
IST is a recently proposed and highly effective technique for solving the aforementioned problems.
We identify fundamental differences between IST and alternative approaches, such as distributed methods with compressed communication.
arXiv Detail & Related papers (2023-06-28T18:14:22Z) - Straightening Out the Straight-Through Estimator: Overcoming
Optimization Challenges in Vector Quantized Networks [35.6604960300194]
This work examines the challenges of training neural networks using vector quantization using straight-through estimation.
We find that a primary cause of training instability is the discrepancy between the model embedding and the code-vector distribution.
We identify the factors that contribute to this issue, including the codebook gradient sparsity and the asymmetric nature of the commitment loss.
arXiv Detail & Related papers (2023-05-15T17:56:36Z) - Probabilistic partition of unity networks for high-dimensional
regression problems [1.0227479910430863]
We explore the partition of unity network (PPOU-Net) model in the context of high-dimensional regression problems.
We propose a general framework focusing on adaptive dimensionality reduction.
The PPOU-Nets consistently outperform the baseline fully-connected neural networks of comparable sizes in numerical experiments.
arXiv Detail & Related papers (2022-10-06T06:01:36Z) - Deep Equilibrium Assisted Block Sparse Coding of Inter-dependent
Signals: Application to Hyperspectral Imaging [71.57324258813675]
A dataset of inter-dependent signals is defined as a matrix whose columns demonstrate strong dependencies.
A neural network is employed to act as structure prior and reveal the underlying signal interdependencies.
Deep unrolling and Deep equilibrium based algorithms are developed, forming highly interpretable and concise deep-learning-based architectures.
arXiv Detail & Related papers (2022-03-29T21:00:39Z) - Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for
sparse recover [87.28082715343896]
We consider deep neural networks for solving inverse problems that are robust to forward model mis-specifications.
We design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem.
The proposed REST network is shown to outperform state-of-the-art model-based and data-driven algorithms in both compressive sensing and radar imaging problems.
arXiv Detail & Related papers (2021-10-20T06:15:45Z) - De-homogenization using Convolutional Neural Networks [1.0323063834827415]
This paper presents a deep learning-based de-homogenization method for structural compliance minimization.
For an appropriate choice of parameters, the de-homogenized designs perform within $7-25%$ of the homogenization-based solution.
arXiv Detail & Related papers (2021-05-10T09:50:06Z) - An AI-Assisted Design Method for Topology Optimization Without
Pre-Optimized Training Data [68.8204255655161]
An AI-assisted design method based on topology optimization is presented, which is able to obtain optimized designs in a direct way.
Designs are provided by an artificial neural network, the predictor, on the basis of boundary conditions and degree of filling as input data.
arXiv Detail & Related papers (2020-12-11T14:33:27Z) - Activation Relaxation: A Local Dynamical Approximation to
Backpropagation in the Brain [62.997667081978825]
Activation Relaxation (AR) is motivated by constructing the backpropagation gradient as the equilibrium point of a dynamical system.
Our algorithm converges rapidly and robustly to the correct backpropagation gradients, requires only a single type of computational unit, and can operate on arbitrary computation graphs.
arXiv Detail & Related papers (2020-09-11T11:56:34Z) - Belief Propagation Reloaded: Learning BP-Layers for Labeling Problems [83.98774574197613]
We take one of the simplest inference methods, a truncated max-product Belief propagation, and add what is necessary to make it a proper component of a deep learning model.
This BP-Layer can be used as the final or an intermediate block in convolutional neural networks (CNNs)
The model is applicable to a range of dense prediction problems, is well-trainable and provides parameter-efficient and robust solutions in stereo, optical flow and semantic segmentation.
arXiv Detail & Related papers (2020-03-13T13:11:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.