Planning to Learn: A Novel Algorithm for Active Learning during Model-Based Planning

• URL: http://arxiv.org/abs/2308.08029v1
• Date: Tue, 15 Aug 2023 20:39:23 GMT
• Title: Planning to Learn: A Novel Algorithm for Active Learning during Model-Based Planning
• Authors: Rowan Hodson, Bruce Bassett, Charel van Hoof, Benjamin Rosman, Mark Solms, Jonathan P. Shock, Ryan Smith
• Abstract summary: We present an extension of SI - sophisticated learning (SL) - that more fully incorporates active learning during planning. SL maintains beliefs about how model parameters would change under the future observations expected under each policy. To accomplish these aims, we make use of a novel, biologically inspired environment designed to highlight the problem structure for which SL offers a unique solution.
• Score: 6.3318086812818475
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Active Inference is a recent framework for modeling planning under uncertainty. Empirical and theoretical work have now begun to evaluate the strengths and weaknesses of this approach and how it might be improved. A recent extension - the sophisticated inference (SI) algorithm - improves performance on multi-step planning problems through recursive decision tree search. However, little work to date has been done to compare SI to other established planning algorithms. SI was also developed with a focus on inference as opposed to learning. The present paper has two aims. First, we compare performance of SI to Bayesian reinforcement learning (RL) schemes designed to solve similar problems. Second, we present an extension of SI - sophisticated learning (SL) - that more fully incorporates active learning during planning. SL maintains beliefs about how model parameters would change under the future observations expected under each policy. This allows a form of counterfactual retrospective inference in which the agent considers what could be learned from current or past observations given different future observations. To accomplish these aims, we make use of a novel, biologically inspired environment designed to highlight the problem structure for which SL offers a unique solution. Here, an agent must continually search for available (but changing) resources in the presence of competing affordances for information gain. Our simulations show that SL outperforms all other algorithms in this context - most notably, Bayes-adaptive RL and upper confidence bound algorithms, which aim to solve multi-step planning problems using similar principles (i.e., directed exploration and counterfactual reasoning). These results provide added support for the utility of Active Inference in solving this class of biologically-relevant problems and offer added tools for testing hypotheses about human cognition.

Related papers

• A Comprehensive Survey on Underwater Image Enhancement Based on Deep Learning [51.7818820745221]
  Underwater image enhancement (UIE) presents a significant challenge within computer vision research. Despite the development of numerous UIE algorithms, a thorough and systematic review is still absent.
  arXiv  Detail & Related papers  (2024-05-30T04:46:40Z)
• Learning-Augmented Algorithms with Explicit Predictors [67.02156211760415]
  Recent advances in algorithmic design show how to utilize predictions obtained by machine learning models from past and present data. Prior research in this context was focused on a paradigm where the predictor is pre-trained on past data and then used as a black box. In this work, we unpack the predictor and integrate the learning problem it gives rise for within the algorithmic challenge.
  arXiv  Detail & Related papers  (2024-03-12T08:40:21Z)
• Representation Learning with Multi-Step Inverse Kinematics: An Efficient and Optimal Approach to Rich-Observation RL [106.82295532402335]
  Existing reinforcement learning algorithms suffer from computational intractability, strong statistical assumptions, and suboptimal sample complexity. We provide the first computationally efficient algorithm that attains rate-optimal sample complexity with respect to the desired accuracy level. Our algorithm, MusIK, combines systematic exploration with representation learning based on multi-step inverse kinematics.
  arXiv  Detail & Related papers  (2023-04-12T14:51:47Z)
• A Survey on the Integration of Machine Learning with Sampling-based Motion Planning [9.264471872135623]
  This survey reviews machine learning efforts to improve the computational efficiency and applicability of Sampling-Based Motion Planners (SBMPs) It first discusses how learning has been used to enhance key components of SBMPs, such as node sampling, collision detection, distance or nearest neighbor, local planning, and termination conditions. It also discusses how machine learning has been used to provide data-driven models of robots, which can then be used by a SBMP.
  arXiv  Detail & Related papers  (2022-11-15T18:13:49Z)
• Hierarchically Structured Task-Agnostic Continual Learning [0.0]
  We take a task-agnostic view of continual learning and develop a hierarchical information-theoretic optimality principle. We propose a neural network layer, called the Mixture-of-Variational-Experts layer, that alleviates forgetting by creating a set of information processing paths. Our approach can operate in a task-agnostic way, i.e., it does not require task-specific knowledge, as is the case with many existing continual learning algorithms.
  arXiv  Detail & Related papers  (2022-11-14T19:53:15Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Latent Properties of Lifelong Learning Systems [59.50307752165016]
  We introduce an algorithm-agnostic explainable surrogate-modeling approach to estimate latent properties of lifelong learning algorithms. We validate the approach for estimating these properties via experiments on synthetic data.
  arXiv  Detail & Related papers  (2022-07-28T20:58:13Z)
• Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms [19.65665942630067]
  Active learning (AL) algorithms may achieve better performance with fewer data because the model guides the data selection process. There is little study on what the optimal AL looks like, which would help researchers understand where their models fall short. We present a simulated annealing algorithm to search for this optimal oracle and analyze it for several tasks.
  arXiv  Detail & Related papers  (2020-12-29T22:56:42Z)
• An Information-Theoretic Framework for Unifying Active Learning Problems [44.758281991246825]
  This paper presents an information-theoretic framework for unifying active learning problems. We first introduce a novel active learning criterion that subsumes an existing LSE algorithm. By exploiting the relationship between LSE and BO, we design a competitive information-theoretic acquisition function for BO.
  arXiv  Detail & Related papers  (2020-12-19T14:22:48Z)
• Discovering Reinforcement Learning Algorithms [53.72358280495428]
  Reinforcement learning algorithms update an agent's parameters according to one of several possible rules. This paper introduces a new meta-learning approach that discovers an entire update rule. It includes both 'what to predict' (e.g. value functions) and 'how to learn from it' by interacting with a set of environments.
  arXiv  Detail & Related papers  (2020-07-17T07:38:39Z)
• A Unifying Framework for Reinforcement Learning and Planning [2.564530030795554]
  This paper presents a unifying algorithmic framework for reinforcement learning and planning (FRAP) At the end of the paper, we compare a variety of well-known planning, model-free and model-based RL algorithms along these dimensions.
  arXiv  Detail & Related papers  (2020-06-26T14:30:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.