Meta-Learning Strategies through Value Maximization in Neural Networks

• URL: http://arxiv.org/abs/2310.19919v2
• Date: Mon, 15 Jul 2024 12:07:03 GMT
• Title: Meta-Learning Strategies through Value Maximization in Neural Networks
• Authors: Rodrigo Carrasco-Davis, Javier Masís, Andrew M. Saxe,
• Abstract summary: We present a learning effort framework capable of efficiently optimizing control signals on a fully normative objective. We apply this framework to investigate the effect of approximations in common meta-learning algorithms. Across settings, we find that control effort is most beneficial when applied to easier aspects of a task early in learning.
• Score: 7.285835869818669
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Biological and artificial learning agents face numerous choices about how to learn, ranging from hyperparameter selection to aspects of task distributions like curricula. Understanding how to make these meta-learning choices could offer normative accounts of cognitive control functions in biological learners and improve engineered systems. Yet optimal strategies remain challenging to compute in modern deep networks due to the complexity of optimizing through the entire learning process. Here we theoretically investigate optimal strategies in a tractable setting. We present a learning effort framework capable of efficiently optimizing control signals on a fully normative objective: discounted cumulative performance throughout learning. We obtain computational tractability by using average dynamical equations for gradient descent, available for simple neural network architectures. Our framework accommodates a range of meta-learning and automatic curriculum learning methods in a unified normative setting. We apply this framework to investigate the effect of approximations in common meta-learning algorithms; infer aspects of optimal curricula; and compute optimal neuronal resource allocation in a continual learning setting. Across settings, we find that control effort is most beneficial when applied to easier aspects of a task early in learning; followed by sustained effort on harder aspects. Overall, the learning effort framework provides a tractable theoretical test bed to study normative benefits of interventions in a variety of learning systems, as well as a formal account of optimal cognitive control strategies over learning trajectories posited by established theories in cognitive neuroscience.

Related papers

• Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training [0.0]
  We develop a dynamic learning rate algorithm that integrates exponential decay and advanced anti-overfitting strategies. We prove that the superlevel sets of the loss function, as influenced by our adaptive learning rate, are always connected.
  arXiv  Detail & Related papers  (2024-09-25T09:27:17Z)
• A Unified Framework for Neural Computation and Learning Over Time [56.44910327178975]
  Hamiltonian Learning is a novel unified framework for learning with neural networks "over time" It is based on differential equations that: (i) can be integrated without the need of external software solvers; (ii) generalize the well-established notion of gradient-based learning in feed-forward and recurrent networks; (iii) open to novel perspectives.
  arXiv  Detail & Related papers  (2024-09-18T14:57:13Z)
• Reasoning Algorithmically in Graph Neural Networks [1.8130068086063336]
  We aim to integrate the structured and rule-based reasoning of algorithms with adaptive learning capabilities of neural networks. This dissertation provides theoretical and practical contributions to this area of research.
  arXiv  Detail & Related papers  (2024-02-21T12:16:51Z)
• Discovering Temporally-Aware Reinforcement Learning Algorithms [42.016150906831776]
  We propose a simple augmentation to two existing objective discovery approaches. We find that commonly used meta-gradient approaches fail to discover adaptive objective functions.
  arXiv  Detail & Related papers  (2024-02-08T17:07:42Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Hierarchically Structured Task-Agnostic Continual Learning [0.0]
  We take a task-agnostic view of continual learning and develop a hierarchical information-theoretic optimality principle. We propose a neural network layer, called the Mixture-of-Variational-Experts layer, that alleviates forgetting by creating a set of information processing paths. Our approach can operate in a task-agnostic way, i.e., it does not require task-specific knowledge, as is the case with many existing continual learning algorithms.
  arXiv  Detail & Related papers  (2022-11-14T19:53:15Z)
• On Data Efficiency of Meta-learning [17.739215706060605]
  We study the often overlooked aspect of the modern meta-learning algorithms -- their data efficiency. We introduce a new simple framework for evaluating meta-learning methods under a limit on the available supervision. We propose active meta-learning, which incorporates active data selection into learning-to-learn, leading to better performance of all methods in the limited supervision regime.
  arXiv  Detail & Related papers  (2021-01-30T01:44:12Z)
• Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms [91.3755431537592]
  We analyze four broad meta-learning strategies which rely on plug-in estimation and pseudo-outcome regression. We highlight how this theoretical reasoning can be used to guide principled algorithm design and translate our analyses into practice.
  arXiv  Detail & Related papers  (2021-01-26T17:11:40Z)
• Importance Weighted Policy Learning and Adaptation [89.46467771037054]
  We study a complementary approach which is conceptually simple, general, modular and built on top of recent improvements in off-policy learning. The framework is inspired by ideas from the probabilistic inference literature and combines robust off-policy learning with a behavior prior. Our approach achieves competitive adaptation performance on hold-out tasks compared to meta reinforcement learning baselines and can scale to complex sparse-reward scenarios.
  arXiv  Detail & Related papers  (2020-09-10T14:16:58Z)
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online [54.15180335046361]
  We propose an algorithm based on meta-gradient descent that discovers its own objective, flexibly parameterised by a deep neural network. Because the objective is discovered online, it can adapt to changes over time. On the Atari Learning Environment, the meta-gradient algorithm adapts over time to learn with greater efficiency.
  arXiv  Detail & Related papers  (2020-07-16T16:17:09Z)
• Provable Representation Learning for Imitation Learning via Bi-level Optimization [60.059520774789654]
  A common strategy in modern learning systems is to learn a representation that is useful for many tasks. We study this strategy in the imitation learning setting for Markov decision processes (MDPs) where multiple experts' trajectories are available. We instantiate this framework for the imitation learning settings of behavior cloning and observation-alone.
  arXiv  Detail & Related papers  (2020-02-24T21:03:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.