An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning

• URL: http://arxiv.org/abs/2101.08390v2
• Date: Mon, 25 Jan 2021 04:55:27 GMT
• Title: An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learning
• Authors: Sharu Theresa Jose and Osvaldo Simeone
• Abstract summary: We present novel information-theoretic bounds on the average absolute value of the meta-generalization gap. Our bounds explicitly capture the impact of task relatedness, the number of tasks, and the number of data samples per task on the meta-generalization gap.
• Score: 44.320945743871285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Meta-learning aims at optimizing the hyperparameters of a model class or training algorithm from the observation of data from a number of related tasks. Following the setting of Baxter [1], the tasks are assumed to belong to the same task environment, which is defined by a distribution over the space of tasks and by per-task data distributions. The statistical properties of the task environment thus dictate the similarity of the tasks. The goal of the meta-learner is to ensure that the hyperparameters obtain a small loss when applied for training of a new task sampled from the task environment. The difference between the resulting average loss, known as meta-population loss, and the corresponding empirical loss measured on the available data from related tasks, known as meta-generalization gap, is a measure of the generalization capability of the meta-learner. In this paper, we present novel information-theoretic bounds on the average absolute value of the meta-generalization gap. Unlike prior work [2], our bounds explicitly capture the impact of task relatedness, the number of tasks, and the number of data samples per task on the meta-generalization gap. Task similarity is gauged via the Kullback-Leibler (KL) and Jensen-Shannon (JS) divergences. We illustrate the proposed bounds on the example of ridge regression with meta-learned bias.

Related papers

• Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
  Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance. We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features. In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
  arXiv  Detail & Related papers  (2024-06-12T08:30:16Z)
• Learning to generate imaginary tasks for improving generalization in meta-learning [12.635773307074022]
  The success of meta-learning on existing benchmarks is predicated on the assumption that the distribution of meta-training tasks covers meta-testing tasks. Recent solutions have pursued augmentation of meta-training tasks, while it is still an open question to generate both correct and sufficiently imaginary tasks. In this paper, we seek an approach that up-samples meta-training tasks from the task representation via a task up-sampling network. Besides, the resulting approach named Adversarial Task Up-sampling (ATU) suffices to generate tasks that can maximally contribute to the latest meta-learner by maximizing an adversarial loss.
  arXiv  Detail & Related papers  (2022-06-09T08:21:05Z)
• Instance-Level Task Parameters: A Robust Multi-task Weighting Framework [17.639472693362926]
  Recent works have shown that deep neural networks benefit from multi-task learning by learning a shared representation across several related tasks. We let the training process dictate the optimal weighting of tasks for every instance in the dataset. We conduct extensive experiments on SURREAL and CityScapes datasets, for human shape and pose estimation, depth estimation and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-11T02:35:42Z)
• Meta-Learning with Fewer Tasks through Task Interpolation [67.03769747726666]
  Current meta-learning algorithms require a large number of meta-training tasks, which may not be accessible in real-world scenarios. By meta-learning with task gradient (MLTI), our approach effectively generates additional tasks by randomly sampling a pair of tasks and interpolating the corresponding features and labels. Empirically, in our experiments on eight datasets from diverse domains, we find that the proposed general MLTI framework is compatible with representative meta-learning algorithms and consistently outperforms other state-of-the-art strategies.
  arXiv  Detail & Related papers  (2021-06-04T20:15:34Z)
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study [75.42182503265056]
  Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm. We deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks.
  arXiv  Detail & Related papers  (2021-05-08T22:26:52Z)
• Transfer Meta-Learning: Information-Theoretic Bounds and Information Meta-Risk Minimization [47.7605527786164]
  Meta-learning automatically infers an inductive bias by observing data from a number of related tasks. We introduce the problem of transfer meta-learning, in which tasks are drawn from a target task environment during meta-testing.
  arXiv  Detail & Related papers  (2020-11-04T12:55:43Z)
• Adaptive Task Sampling for Meta-Learning [79.61146834134459]
  Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time. We propose an adaptive task sampling method to improve the generalization performance.
  arXiv  Detail & Related papers  (2020-07-17T03:15:53Z)
• Information-Theoretic Generalization Bounds for Meta-Learning and Applications [42.275148861039895]
  Key performance measure for meta-learning is the meta-generalization gap. This paper presents novel information-theoretic upper bounds on the meta-generalization gap.
  arXiv  Detail & Related papers  (2020-05-09T05:48:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.