MARS: Meta-Learning as Score Matching in the Function Space
- URL: http://arxiv.org/abs/2210.13319v3
- Date: Sat, 10 Jun 2023 10:11:33 GMT
- Title: MARS: Meta-Learning as Score Matching in the Function Space
- Authors: Krunoslav Lehman Pavasovic, Jonas Rothfuss and Andreas Krause
- Abstract summary: We present a novel approach to extracting inductive biases from a set of related datasets.
We use functional Bayesian neural network inference, which views the prior as a process and performs inference in the function space.
Our approach can seamlessly acquire and represent complex prior knowledge by metalearning the score function of the data-generating process.
- Score: 79.73213540203389
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Meta-learning aims to extract useful inductive biases from a set of related
datasets. In Bayesian meta-learning, this is typically achieved by constructing
a prior distribution over neural network parameters. However, specifying
families of computationally viable prior distributions over the
high-dimensional neural network parameters is difficult. As a result, existing
approaches resort to meta-learning restrictive diagonal Gaussian priors,
severely limiting their expressiveness and performance. To circumvent these
issues, we approach meta-learning through the lens of functional Bayesian
neural network inference, which views the prior as a stochastic process and
performs inference in the function space. Specifically, we view the
meta-training tasks as samples from the data-generating process and formalize
meta-learning as empirically estimating the law of this stochastic process. Our
approach can seamlessly acquire and represent complex prior knowledge by
meta-learning the score function of the data-generating process marginals
instead of parameter space priors. In a comprehensive benchmark, we demonstrate
that our method achieves state-of-the-art performance in terms of predictive
accuracy and substantial improvements in the quality of uncertainty estimates.
Related papers
- Deep Learning and genetic algorithms for cosmological Bayesian inference speed-up [0.0]
We present a novel approach to accelerate the Bayesian inference process, focusing specifically on the nested sampling algorithms.
Our proposed method utilizes the power of deep learning, employing feedforward neural networks to approximate the likelihood function dynamically during the Bayesian inference process.
The implementation integrates with nested sampling algorithms and has been thoroughly evaluated using both simple cosmological dark energy models and diverse observational datasets.
arXiv Detail & Related papers (2024-05-06T09:14:58Z) - Rethinking of Encoder-based Warm-start Methods in Hyperparameter Optimization [0.0]
We introduce a new approach for representation learning on tabular data based on Tomoharu Iwata and Atsutoshi Kumagai.
We show that general representations may not suffice for some meta-tasks where requirements are not explicitly considered during extraction.
arXiv Detail & Related papers (2024-03-07T18:16:29Z) - Function-Space Regularization in Neural Networks: A Probabilistic
Perspective [51.133793272222874]
We show that we can derive a well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training.
We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection and highly-calibrated predictive uncertainty estimates.
arXiv Detail & Related papers (2023-12-28T17:50:56Z) - End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures.
We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
arXiv Detail & Related papers (2023-05-25T10:58:46Z) - Scalable Bayesian Meta-Learning through Generalized Implicit Gradients [64.21628447579772]
Implicit Bayesian meta-learning (iBaML) method broadens the scope of learnable priors, but also quantifies the associated uncertainty.
Analytical error bounds are established to demonstrate the precision and efficiency of the generalized implicit gradient over the explicit one.
arXiv Detail & Related papers (2023-03-31T02:10:30Z) - Memory-Based Meta-Learning on Non-Stationary Distributions [29.443692147512742]
Memory-based meta-learning is a technique for approximating Bayes-optimal predictors.
We show that memory-based neural models, including Transformers, LSTMs, and RNNs can learn to accurately approximate known Bayes-optimal algorithms.
arXiv Detail & Related papers (2023-02-06T19:08:59Z) - Learning Large-scale Neural Fields via Context Pruned Meta-Learning [60.93679437452872]
We introduce an efficient optimization-based meta-learning technique for large-scale neural field training.
We show how gradient re-scaling at meta-test time allows the learning of extremely high-quality neural fields.
Our framework is model-agnostic, intuitive, straightforward to implement, and shows significant reconstruction improvements for a wide range of signals.
arXiv Detail & Related papers (2023-02-01T17:32:16Z) - Bayesian Meta-Learning Through Variational Gaussian Processes [0.0]
We extend Gaussian-process-based meta-learning to allow for high-quality, arbitrary non-Gaussian uncertainty predictions.
Our method performs significantly better than existing Bayesian meta-learning baselines.
arXiv Detail & Related papers (2021-10-21T10:44:23Z) - Meta-Learning Reliable Priors in the Function Space [36.869587157481284]
We introduce a novel meta-learning framework, called F-PACOH, that treats meta-learned priors as processes and performs meta-level regularization directly in the function space.
This allows us to directly steer the predictions of the meta-learner towards high uncertainty in regions of insufficient meta-training data and, thus, obtain well-calibrated uncertainty estimates.
arXiv Detail & Related papers (2021-06-06T18:07:49Z) - PACOH: Bayes-Optimal Meta-Learning with PAC-Guarantees [77.67258935234403]
We provide a theoretical analysis using the PAC-Bayesian framework and derive novel generalization bounds for meta-learning.
We develop a class of PAC-optimal meta-learning algorithms with performance guarantees and a principled meta-level regularization.
arXiv Detail & Related papers (2020-02-13T15:01:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.