Learning Abstract Task Representations
- URL: http://arxiv.org/abs/2101.07852v3
- Date: Thu, 28 Jan 2021 20:07:20 GMT
- Title: Learning Abstract Task Representations
- Authors: Mikhail M. Meskhi, Adriano Rivolli, Rafael G. Mantovani, Ricardo
Vilalta
- Abstract summary: We propose a method to induce new abstract meta-features as latent variables in a deep neural network.
We demonstrate our methodology using a deep neural network as a feature extractor.
- Score: 0.6690874707758511
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A proper form of data characterization can guide the process of
learning-algorithm selection and model-performance estimation. The field of
meta-learning has provided a rich body of work describing effective forms of
data characterization using different families of meta-features (statistical,
model-based, information-theoretic, topological, etc.). In this paper, we start
with the abundant set of existing meta-features and propose a method to induce
new abstract meta-features as latent variables in a deep neural network. We
discuss the pitfalls of using traditional meta-features directly and argue for
the importance of learning high-level task properties. We demonstrate our
methodology using a deep neural network as a feature extractor. We demonstrate
that 1) induced meta-models mapping abstract meta-features to generalization
performance outperform other methods by ~18% on average, and 2) abstract
meta-features attain high feature-relevance scores.
Related papers
- Learning Task Representations from In-Context Learning [73.72066284711462]
Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning.
We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads.
We show that our method's effectiveness stems from aligning the distribution of the last hidden state with that of an optimally performing in-context-learned model.
arXiv Detail & Related papers (2025-02-08T00:16:44Z) - Dynamics of Meta-learning Representation in the Teacher-student Scenario [8.099691748821114]
We investigate the meta-learning dynamics of nonlinear two-layer neural networks trained on streaming tasks in a teacher-student scenario.
We characterize the macroscopic behavior of the meta-training processes, the formation of the shared representation, and the generalization ability of the model on new tasks.
arXiv Detail & Related papers (2024-08-22T16:59:32Z) - Automatic learning algorithm selection for classification via
convolutional neural networks [0.0]
The goal of this study is to learn the inherent structure of the data without identifying meta-features.
Experiments with simulated datasets show that the proposed approach achieves nearly perfect performance in identifying linear and nonlinear patterns.
arXiv Detail & Related papers (2023-05-16T01:57:01Z) - Deep networks for system identification: a Survey [56.34005280792013]
System identification learns mathematical descriptions of dynamic systems from input-output data.
Main aim of the identified model is to predict new data from previous observations.
We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
arXiv Detail & Related papers (2023-01-30T12:38:31Z) - MARS: Meta-Learning as Score Matching in the Function Space [79.73213540203389]
We present a novel approach to extracting inductive biases from a set of related datasets.
We use functional Bayesian neural network inference, which views the prior as a process and performs inference in the function space.
Our approach can seamlessly acquire and represent complex prior knowledge by metalearning the score function of the data-generating process.
arXiv Detail & Related papers (2022-10-24T15:14:26Z) - Meta-Learning with Self-Improving Momentum Target [72.98879709228981]
We propose Self-improving Momentum Target (SiMT) to improve the performance of a meta-learner.
SiMT generates the target model by adapting from the temporal ensemble of the meta-learner.
We show that SiMT brings a significant performance gain when combined with a wide range of meta-learning methods.
arXiv Detail & Related papers (2022-10-11T06:45:15Z) - Towards Open-World Feature Extrapolation: An Inductive Graph Learning
Approach [80.8446673089281]
We propose a new learning paradigm with graph representation and learning.
Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
arXiv Detail & Related papers (2021-10-09T09:02:45Z) - Meta-Learning with Fewer Tasks through Task Interpolation [67.03769747726666]
Current meta-learning algorithms require a large number of meta-training tasks, which may not be accessible in real-world scenarios.
By meta-learning with task gradient (MLTI), our approach effectively generates additional tasks by randomly sampling a pair of tasks and interpolating the corresponding features and labels.
Empirically, in our experiments on eight datasets from diverse domains, we find that the proposed general MLTI framework is compatible with representative meta-learning algorithms and consistently outperforms other state-of-the-art strategies.
arXiv Detail & Related papers (2021-06-04T20:15:34Z) - Predicting Scores of Medical Imaging Segmentation Methods with
Meta-Learning [0.30458514384586394]
We investigate meta-learning for segmentation across ten datasets of different organs and modalities.
We use support vector regression and deep neural networks to learn the relationship between the meta-features and prior model performance.
These results demonstrate the potential of meta-learning in medical imaging.
arXiv Detail & Related papers (2020-05-08T07:47:52Z) - Towards explainable meta-learning [5.802346990263708]
Meta-learning aims at discovering how different machine learning algorithms perform on a wide range of predictive tasks.
State of the art approaches are focused on searching for the best meta-model but do not explain how these different aspects contribute to its performance.
We propose techniques developed for eXplainable Artificial Intelligence (XAI) to examine and extract knowledge from black-box surrogate models.
arXiv Detail & Related papers (2020-02-11T09:42:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.