Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling

• URL: http://arxiv.org/abs/2207.04179v1
• Date: Sat, 9 Jul 2022 02:28:58 GMT
• Title: Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling
• Authors: Tung Nguyen and Aditya Grover
• Abstract summary: We propose Transformer Neural Processes (TNPs) for uncertainty-aware meta learning. We learn TNPs via an autoregressive likelihood-based objective and instantiate it with a novel transformer-based architecture. We show that TNPs achieve state-of-the-art performance on various benchmark problems.
• Score: 26.377099481072992
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Processes (NPs) are a popular class of approaches for meta-learning. Similar to Gaussian Processes (GPs), NPs define distributions over functions and can estimate uncertainty in their predictions. However, unlike GPs, NPs and their variants suffer from underfitting and often have intractable likelihoods, which limit their applications in sequential decision making. We propose Transformer Neural Processes (TNPs), a new member of the NP family that casts uncertainty-aware meta learning as a sequence modeling problem. We learn TNPs via an autoregressive likelihood-based objective and instantiate it with a novel transformer-based architecture. The model architecture respects the inductive biases inherent to the problem structure, such as invariance to the observed data points and equivariance to the unobserved points. We further investigate knobs within the TNP framework that tradeoff expressivity of the decoding distribution with extra computation. Empirically, we show that TNPs achieve state-of-the-art performance on various benchmark problems, outperforming all previous NP variants on meta regression, image completion, contextual multi-armed bandits, and Bayesian optimization.

Related papers

• In-Context In-Context Learning with Transformer Neural Processes [50.57807892496024]
  We develop the in-context in-context learning pseudo-token TNP (ICICL-TNP) The ICICL-TNP is capable of conditioning on both sets of datapoints and sets of datasets, enabling it to perform in-context in-context learning. We demonstrate the importance of in-context in-context learning and the effectiveness of the ICICL-TNP in a number of experiments.
  arXiv  Detail & Related papers  (2024-06-19T12:26:36Z)
• Translation Equivariant Transformer Neural Processes [22.463975744505717]
  The effectiveness of neural processes (NPs) in modelling posterior prediction maps has significantly improved since their inception. This improvement can be attributed to two principal factors: (1) advancements in the architecture of permutation invariant set functions, which are to all distributionss; and (2) leveraging symmetries present in the true posterior predictive map.
  arXiv  Detail & Related papers  (2024-06-18T08:58:59Z)
• Spectral Convolutional Conditional Neural Processes [4.52069311861025]
  Conditional Neural Processes (CNPs) constitute a family of probabilistic models that harness the flexibility of neural networks to parameterize processes. We propose Spectral Convolutional Conditional Neural Processes (SConvCNPs), a new addition to the NPs family that allows for more efficient representation of functions in the frequency domain.
  arXiv  Detail & Related papers  (2024-04-19T21:13:18Z)
• Deep Stochastic Processes via Functional Markov Transition Operators [59.55961312230447]
  We introduce a new class of Processes (SPs) constructed by stacking sequences of neural parameterised Markov transition operators in function space. We prove that these Markov transition operators can preserve the exchangeability and consistency of SPs.
  arXiv  Detail & Related papers  (2023-05-24T21:15:23Z)
• Versatile Neural Processes for Learning Implicit Neural Representations [57.090658265140384]
  We propose Versatile Neural Processes (VNP), which largely increases the capability of approximating functions. Specifically, we introduce a bottleneck encoder that produces fewer and informative context tokens, relieving the high computational cost. We demonstrate the effectiveness of the proposed VNP on a variety of tasks involving 1D, 2D and 3D signals.
  arXiv  Detail & Related papers  (2023-01-21T04:08:46Z)
• Latent Bottlenecked Attentive Neural Processes [71.18817592128207]
  We present Latent Bottlenecked Attentive Neural Processes (LBANPs) LBANPs have a querying computational complexity independent of the number of context datapoints. We show LBANPs achieve results competitive with the state-of-the-art on meta-regression, image completion, and contextual multi-armed bandits.
  arXiv  Detail & Related papers  (2022-11-15T19:21:41Z)
• The Gaussian Neural Process [39.81327564209865]
  We provide a rigorous analysis of the standard maximum-likelihood objective used to train conditional NPs. We propose a new member to the Neural Process family called the Neural Process (GNP), which models predictive correlations, incorporates translation, provides universal approximation guarantees, and demonstrates encouraging performance.
  arXiv  Detail & Related papers  (2021-01-10T19:15:27Z)
• Doubly Stochastic Variational Inference for Neural Processes with Hierarchical Latent Variables [37.43541345780632]
  We present a new variant of Neural Process (NP) model that we call Doubly Variational Neural Process (DSVNP) This model combines the global latent variable and local latent variables for prediction. We evaluate this model in several experiments, and our results demonstrate competitive prediction performance in multi-output regression and uncertainty estimation in classification.
  arXiv  Detail & Related papers  (2020-08-21T13:32:12Z)
• Bootstrapping Neural Processes [114.97111530885093]
  Neural Processes (NPs) implicitly define a broad class of processes with neural networks. NPs still rely on an assumption that uncertainty in processes is modeled by a single latent variable. We propose the Boostrapping Neural Process (BNP), a novel extension of the NP family using the bootstrap.
  arXiv  Detail & Related papers  (2020-08-07T02:23:34Z)
• Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes [32.02612871707347]
  We propose ConvNP, which endows Neural Processes (NPs) with translation equivariance and extends convolutional conditional NPs to allow for dependencies in the predictive distribution. We demonstrate the strong performance and generalization capabilities of ConvNPs on 1D, regression image completion, and various tasks with real-world-temporal data.
  arXiv  Detail & Related papers  (2020-07-02T18:25:27Z)
• NP-PROV: Neural Processes with Position-Relevant-Only Variances [113.20013269514327]
  We present a new member named Neural Processes with Position-Relevant-Only Variances (NP-PROV) NP-PROV hypothesizes that a target point close to a context point has small uncertainty, regardless of the function value at that position. Our evaluation on synthetic and real-world datasets reveals that NP-PROV can achieve state-of-the-art likelihood while retaining a bounded variance.
  arXiv  Detail & Related papers  (2020-06-15T06:11:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.