Kolmogorov n-Widths for Multitask Physics-Informed Machine Learning
(PIML) Methods: Towards Robust Metrics
- URL: http://arxiv.org/abs/2402.11126v1
- Date: Fri, 16 Feb 2024 23:21:40 GMT
- Title: Kolmogorov n-Widths for Multitask Physics-Informed Machine Learning
(PIML) Methods: Towards Robust Metrics
- Authors: Michael Penwarden, Houman Owhadi, Robert M. Kirby
- Abstract summary: This topic encompasses a broad array of methods and models aimed at solving a single or a collection of PDE problems, called multitask learning.
PIML is characterized by the incorporation of physical laws into the training process of machine learning models in lieu of large data when solving PDE problems.
- Score: 10.005376908536178
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Physics-informed machine learning (PIML) as a means of solving partial
differential equations (PDE) has garnered much attention in the Computational
Science and Engineering (CS&E) world. This topic encompasses a broad array of
methods and models aimed at solving a single or a collection of PDE problems,
called multitask learning. PIML is characterized by the incorporation of
physical laws into the training process of machine learning models in lieu of
large data when solving PDE problems. Despite the overall success of this
collection of methods, it remains incredibly difficult to analyze, benchmark,
and generally compare one approach to another. Using Kolmogorov n-widths as a
measure of effectiveness of approximating functions, we judiciously apply this
metric in the comparison of various multitask PIML architectures. We compute
lower accuracy bounds and analyze the model's learned basis functions on
various PDE problems. This is the first objective metric for comparing
multitask PIML architectures and helps remove uncertainty in model validation
from selective sampling and overfitting. We also identify avenues of
improvement for model architectures, such as the choice of activation function,
which can drastically affect model generalization to "worst-case" scenarios,
which is not observed when reporting task-specific errors. We also incorporate
this metric into the optimization process through regularization, which
improves the models' generalizability over the multitask PDE problem.
Related papers
- Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion [53.33473557562837]
Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost.
We propose a practical and scalable approach to solve this problem via mixture of experts (MoE) based model fusion.
By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives.
arXiv Detail & Related papers (2024-06-14T07:16:18Z) - MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation [80.47072100963017]
Model merging is an effective approach to combine multiple single-task models, fine-tuned from the same pre-trained model, into a multitask model.
Existing model-merging methods focus on enhancing average task accuracy.
We introduce a novel low-compute algorithm, Model Merging with Amortized Pareto Front (MAP)
arXiv Detail & Related papers (2024-06-11T17:55:25Z) - UCB-driven Utility Function Search for Multi-objective Reinforcement Learning [75.11267478778295]
In Multi-objective Reinforcement Learning (MORL) agents are tasked with optimising decision-making behaviours.
We focus on the case of linear utility functions parameterised by weight vectors w.
We introduce a method based on Upper Confidence Bound to efficiently search for the most promising weight vectors during different stages of the learning process.
arXiv Detail & Related papers (2024-05-01T09:34:42Z) - Sample Complexity Characterization for Linear Contextual MDPs [67.79455646673762]
Contextual decision processes (CMDPs) describe a class of reinforcement learning problems in which the transition kernels and reward functions can change over time with different MDPs indexed by a context variable.
CMDPs serve as an important framework to model many real-world applications with time-varying environments.
We study CMDPs under two linear function approximation models: Model I with context-varying representations and common linear weights for all contexts; and Model II with common representations for all contexts and context-varying linear weights.
arXiv Detail & Related papers (2024-02-05T03:25:04Z) - Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently.
Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable.
We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
arXiv Detail & Related papers (2024-02-01T08:58:57Z) - Meta-Learning for Airflow Simulations with Graph Neural Networks [3.52359746858894]
We present a meta-learning approach to enhance the performance of learned models on out-of-distribution (OoD) samples.
Specifically, we set the airflow simulation in CFD over various airfoils as a meta-learning problem, where each set of examples defined on a single airfoil shape is treated as a separate task.
We experimentally demonstrate the efficiency of the proposed approach for improving the OoD generalization performance of learned models.
arXiv Detail & Related papers (2023-06-18T19:25:13Z) - Multi-Task Learning on Networks [0.0]
Multi-objective optimization problems arising in the multi-task learning context have specific features and require adhoc methods.
In this thesis the solutions in the Input Space are represented as probability distributions encapsulating the knowledge contained in the function evaluations.
In this space of probability distributions, endowed with the metric given by the Wasserstein distance, a new algorithm MOEA/WST can be designed in which the model is not directly on the objective function.
arXiv Detail & Related papers (2021-12-07T09:13:10Z) - A composable autoencoder-based iterative algorithm for accelerating
numerical simulations [0.0]
CoAE-MLSim is an unsupervised, lower-dimensional, local method that is motivated from key ideas used in commercial PDE solvers.
It is tested for a variety of complex engineering cases to demonstrate its computational speed, accuracy, scalability, and generalization across different PDE conditions.
arXiv Detail & Related papers (2021-10-07T20:22:37Z) - Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems.
Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC.
We provide a practical parametrization of the core optimization problem.
arXiv Detail & Related papers (2021-07-08T18:01:02Z) - Solver-in-the-Loop: Learning from Differentiable Physics to Interact
with Iterative PDE-Solvers [26.444103444634994]
We show that machine learning can improve the solution accuracy by correcting for effects not captured by the discretized PDE.
We find that previously used learning approaches are significantly outperformed by methods that integrate the solver into the training loop.
This provides the model with realistic input distributions that take previous corrections into account.
arXiv Detail & Related papers (2020-06-30T18:00:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.