Multi-Task Learning on Networks
- URL: http://arxiv.org/abs/2112.04891v1
- Date: Tue, 7 Dec 2021 09:13:10 GMT
- Title: Multi-Task Learning on Networks
- Authors: Andrea Ponti
- Abstract summary: Multi-objective optimization problems arising in the multi-task learning context have specific features and require adhoc methods.
In this thesis the solutions in the Input Space are represented as probability distributions encapsulating the knowledge contained in the function evaluations.
In this space of probability distributions, endowed with the metric given by the Wasserstein distance, a new algorithm MOEA/WST can be designed in which the model is not directly on the objective function.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The multi-task learning (MTL) paradigm can be traced back to an early paper
of Caruana (1997) in which it was argued that data from multiple tasks can be
used with the aim to obtain a better performance over learning each task
independently. A solution of MTL with conflicting objectives requires modelling
the trade-off among them which is generally beyond what a straight linear
combination can achieve. A theoretically principled and computationally
effective strategy is finding solutions which are not dominated by others as it
is addressed in the Pareto analysis. Multi-objective optimization problems
arising in the multi-task learning context have specific features and require
adhoc methods. The analysis of these features and the proposal of a new
computational approach represent the focus of this work. Multi-objective
evolutionary algorithms (MOEAs) can easily include the concept of dominance and
therefore the Pareto analysis. The major drawback of MOEAs is a low sample
efficiency with respect to function evaluations. The key reason for this
drawback is that most of the evolutionary approaches do not use models for
approximating the objective function. Bayesian Optimization takes a radically
different approach based on a surrogate model, such as a Gaussian Process. In
this thesis the solutions in the Input Space are represented as probability
distributions encapsulating the knowledge contained in the function
evaluations. In this space of probability distributions, endowed with the
metric given by the Wasserstein distance, a new algorithm MOEA/WST can be
designed in which the model is not directly on the objective function but in an
intermediate Information Space where the objects from the input space are
mapped into histograms. Computational results show that the sample efficiency
and the quality of the Pareto set provided by MOEA/WST are significantly better
than in the standard MOEA.
Related papers
- Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion [53.33473557562837]
Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost.
We propose a practical and scalable approach to solve this problem via mixture of experts (MoE) based model fusion.
By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives.
arXiv Detail & Related papers (2024-06-14T07:16:18Z) - Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance.
We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features.
In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
arXiv Detail & Related papers (2024-06-12T08:30:16Z) - MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation [80.47072100963017]
Model merging is an effective approach to combine multiple single-task models, fine-tuned from the same pre-trained model, into a multitask model.
Existing model-merging methods focus on enhancing average task accuracy.
We introduce a novel low-compute algorithm, Model Merging with Amortized Pareto Front (MAP)
arXiv Detail & Related papers (2024-06-11T17:55:25Z) - Kolmogorov n-Widths for Multitask Physics-Informed Machine Learning
(PIML) Methods: Towards Robust Metrics [10.005376908536178]
This topic encompasses a broad array of methods and models aimed at solving a single or a collection of PDE problems, called multitask learning.
PIML is characterized by the incorporation of physical laws into the training process of machine learning models in lieu of large data when solving PDE problems.
arXiv Detail & Related papers (2024-02-16T23:21:40Z) - End-to-End Pareto Set Prediction with Graph Neural Networks for
Multi-objective Facility Location [10.130342722193204]
Facility location problems (FLPs) are a typical class of NP-hard optimization problems, which are widely seen in the supply chain and logistics.
In this paper, we consider the multi-objective facility location problem (MO-FLP) that simultaneously minimizes the overall cost and maximizes the system reliability.
Two graph neural networks are constructed to learn the implicit graph representation on nodes and edges.
arXiv Detail & Related papers (2022-10-27T07:15:55Z) - Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems.
Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC.
We provide a practical parametrization of the core optimization problem.
arXiv Detail & Related papers (2021-07-08T18:01:02Z) - MAML is a Noisy Contrastive Learner [72.04430033118426]
Model-agnostic meta-learning (MAML) is one of the most popular and widely-adopted meta-learning algorithms nowadays.
We provide a new perspective to the working mechanism of MAML and discover that: MAML is analogous to a meta-learner using a supervised contrastive objective function.
We propose a simple but effective technique, zeroing trick, to alleviate such interference.
arXiv Detail & Related papers (2021-06-29T12:52:26Z) - Offline Model-Based Optimization via Normalized Maximum Likelihood
Estimation [101.22379613810881]
We consider data-driven optimization problems where one must maximize a function given only queries at a fixed set of points.
This problem setting emerges in many domains where function evaluation is a complex and expensive process.
We propose a tractable approximation that allows us to scale our method to high-capacity neural network models.
arXiv Detail & Related papers (2021-02-16T06:04:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.