Optimal Transport-inspired Deep Learning Framework for Slow-Decaying
Problems: Exploiting Sinkhorn Loss and Wasserstein Kernel
- URL: http://arxiv.org/abs/2308.13840v1
- Date: Sat, 26 Aug 2023 10:24:43 GMT
- Title: Optimal Transport-inspired Deep Learning Framework for Slow-Decaying
Problems: Exploiting Sinkhorn Loss and Wasserstein Kernel
- Authors: Moaad Khamlich and Federico Pichi and Gianluigi Rozza
- Abstract summary: Reduced order models (ROMs) are widely used in scientific computing to tackle high-dimensional systems.
We propose a novel ROM framework that integrates optimal transport theory and neural network-based methods.
Our framework outperforms traditional ROM methods in terms of accuracy and computational efficiency.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Reduced order models (ROMs) are widely used in scientific computing to tackle
high-dimensional systems. However, traditional ROM methods may only partially
capture the intrinsic geometric characteristics of the data. These
characteristics encompass the underlying structure, relationships, and
essential features crucial for accurate modeling.
To overcome this limitation, we propose a novel ROM framework that integrates
optimal transport (OT) theory and neural network-based methods. Specifically,
we investigate the Kernel Proper Orthogonal Decomposition (kPOD) method
exploiting the Wasserstein distance as the custom kernel, and we efficiently
train the resulting neural network (NN) employing the Sinkhorn algorithm. By
leveraging an OT-based nonlinear reduction, the presented framework can capture
the geometric structure of the data, which is crucial for accurate learning of
the reduced solution manifold. When compared with traditional metrics such as
mean squared error or cross-entropy, exploiting the Sinkhorn divergence as the
loss function enhances stability during training, robustness against
overfitting and noise, and accelerates convergence.
To showcase the approach's effectiveness, we conduct experiments on a set of
challenging test cases exhibiting a slow decay of the Kolmogorov n-width. The
results show that our framework outperforms traditional ROM methods in terms of
accuracy and computational efficiency.
Related papers
- Efficient kernel surrogates for neural network-based regression [0.8030359871216615]
We study the performance of the Conjugate Kernel (CK), an efficient approximation to the Neural Tangent Kernel (NTK)
We show that the CK performance is only marginally worse than that of the NTK and, in certain cases, is shown to be superior.
In addition to providing a theoretical grounding for using CKs instead of NTKs, our framework suggests a recipe for improving DNN accuracy inexpensively.
arXiv Detail & Related papers (2023-10-28T06:41:47Z) - Large-Scale OD Matrix Estimation with A Deep Learning Method [70.78575952309023]
The proposed method integrates deep learning and numerical optimization algorithms to infer matrix structure and guide numerical optimization.
We conducted tests to demonstrate the good generalization performance of our method on a large-scale synthetic dataset.
arXiv Detail & Related papers (2023-10-09T14:30:06Z) - Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels [57.46832672991433]
We propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS)
We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises.
We develop an expectation-propagation expectation-maximization algorithm for efficient posterior inference and function estimation.
arXiv Detail & Related papers (2023-10-09T03:55:09Z) - An Optimization-based Deep Equilibrium Model for Hyperspectral Image
Deconvolution with Convergence Guarantees [71.57324258813675]
We propose a novel methodology for addressing the hyperspectral image deconvolution problem.
A new optimization problem is formulated, leveraging a learnable regularizer in the form of a neural network.
The derived iterative solver is then expressed as a fixed-point calculation problem within the Deep Equilibrium framework.
arXiv Detail & Related papers (2023-06-10T08:25:16Z) - A Kernel-Expanded Stochastic Neural Network [10.837308632004644]
Deep neural network often gets trapped into a local minimum in training.
New kernel-expanded neural network (K-StoNet) model reformulates the network as a latent variable model.
Model can be easily trained using the imputationregularized optimization (IRO) algorithm.
arXiv Detail & Related papers (2022-01-14T06:42:42Z) - Efficient Micro-Structured Weight Unification and Pruning for Neural
Network Compression [56.83861738731913]
Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices.
Previous unstructured or structured weight pruning methods can hardly truly accelerate inference.
We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
arXiv Detail & Related papers (2021-06-15T17:22:59Z) - Level-Set Curvature Neural Networks: A Hybrid Approach [0.0]
We present a hybrid strategy based on deep learning to compute mean curvature in the level-set method.
The proposed inference system combines a dictionary of improved regression models with standard numerical schemes to estimate curvature more accurately.
Our findings confirm that machine learning is a promising venue for devising viable solutions to the level-set method's numerical shortcomings.
arXiv Detail & Related papers (2021-04-07T06:51:52Z) - Random Features for the Neural Tangent Kernel [57.132634274795066]
We propose an efficient feature map construction of the Neural Tangent Kernel (NTK) of fully-connected ReLU network.
We show that dimension of the resulting features is much smaller than other baseline feature map constructions to achieve comparable error bounds both in theory and practice.
arXiv Detail & Related papers (2021-04-03T09:08:12Z) - Physics Informed Deep Kernel Learning [24.033468062984458]
Physics Informed Deep Kernel Learning (PI-DKL) exploits physics knowledge represented by differential equations with latent sources.
For efficient and effective inference, we marginalize out the latent variables and derive a collapsed model evidence lower bound (ELBO)
arXiv Detail & Related papers (2020-06-08T22:43:31Z) - Neural Control Variates [71.42768823631918]
We show that a set of neural networks can face the challenge of finding a good approximation of the integrand.
We derive a theoretically optimal, variance-minimizing loss function, and propose an alternative, composite loss for stable online training in practice.
Specifically, we show that the learned light-field approximation is of sufficient quality for high-order bounces, allowing us to omit the error correction and thereby dramatically reduce the noise at the cost of negligible visible bias.
arXiv Detail & Related papers (2020-06-02T11:17:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.