Learning Universal Predictors

• URL: http://arxiv.org/abs/2401.14953v1
• Date: Fri, 26 Jan 2024 15:37:16 GMT
• Title: Learning Universal Predictors
• Authors: Jordi Grau-Moya, Tim Genewein, Marcus Hutter, Laurent Orseau, Gr\'egoire Del\'etang, Elliot Catt, Anian Ruoss, Li Kevin Wenliang, Christopher Mattern, Matthew Aitchison, Joel Veness
• Abstract summary: We explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.
• Score: 23.18743879588599
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Meta-learning has emerged as a powerful approach to train neural networks to learn new tasks quickly from limited data. Broad exposure to different tasks leads to versatile representations enabling general problem solving. But, what are the limits of meta-learning? In this work, we explore the potential of amortizing the most powerful universal predictor, namely Solomonoff Induction (SI), into neural networks via leveraging meta-learning to its limits. We use Universal Turing Machines (UTMs) to generate training data used to expose networks to a broad range of patterns. We provide theoretical analysis of the UTM data generation processes and meta-training protocols. We conduct comprehensive experiments with neural architectures (e.g. LSTMs, Transformers) and algorithmic data generators of varying complexity and universality. Our results suggest that UTM data is a valuable resource for meta-learning, and that it can be used to train neural networks capable of learning universal prediction strategies.

Related papers

• Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
  We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations. Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
  arXiv  Detail & Related papers  (2024-02-20T15:23:24Z)
• aSTDP: A More Biologically Plausible Learning [0.0]
  We introduce approximate STDP, a new neural networks learning framework. It uses only STDP rules for supervised and unsupervised learning. It can make predictions or generate patterns in one model without additional configuration.
  arXiv  Detail & Related papers  (2022-05-22T08:12:50Z)
• PMFL: Partial Meta-Federated Learning for heterogeneous tasks and its applications on real-world medical records [11.252157002705484]
  Federated machine learning is a versatile and flexible tool to utilize distributed data from different sources. We propose a new algorithm, which is an integration of federated learning and meta-learning, to tackle this issue. We show that our algorithm could obtain the fastest training speed and achieve the best performance when dealing with heterogeneous medical datasets.
  arXiv  Detail & Related papers  (2021-12-10T03:55:03Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)
• Generalising via Meta-Examples for Continual Learning in the Wild [24.09600678738403]
  We develop a novel strategy to deal with neural networks that "learn in the wild" We equip it with MEML - Meta-Example Meta-Learning - a new module that simultaneously alleviates catastrophic forgetting. We extend it by adopting a technique that creates various augmented tasks and optimises over the hardest.
  arXiv  Detail & Related papers  (2021-01-28T15:51:54Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• Neural Complexity Measures [96.06344259626127]
  We propose Neural Complexity (NC), a meta-learning framework for predicting generalization. Our model learns a scalar complexity measure through interactions with many heterogeneous tasks in a data-driven way.
  arXiv  Detail & Related papers  (2020-08-07T02:12:10Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding [97.85957811603251]
  We present MT-DNN, an open-source natural language understanding (NLU) toolkit that makes it easy for researchers and developers to train customized deep learning models. Built upon PyTorch and Transformers, MT-DNN is designed to facilitate rapid customization for a broad spectrum of NLU tasks. A unique feature of MT-DNN is its built-in support for robust and transferable learning using the adversarial multi-task learning paradigm.
  arXiv  Detail & Related papers  (2020-02-19T03:05:28Z)
• Federated Learning with Matched Averaging [43.509797844077426]
  Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures.
  arXiv  Detail & Related papers  (2020-02-15T20:09:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.