Value of Information and Reward Specification in Active Inference and POMDPs

• URL: http://arxiv.org/abs/2408.06542v1
• Date: Tue, 13 Aug 2024 00:32:05 GMT
• Title: Value of Information and Reward Specification in Active Inference and POMDPs
• Authors: Ran Wei,
• Abstract summary: Expected free energy (EFE) is a central quantity in active inference. We show that EFE approximates the Bayes optimal RL policy via information value.
• Score: 7.120454740315046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Expected free energy (EFE) is a central quantity in active inference which has recently gained popularity due to its intuitive decomposition of the expected value of control into a pragmatic and an epistemic component. While numerous conjectures have been made to justify EFE as a decision making objective function, the most widely accepted is still its intuitiveness and resemblance to variational free energy in approximate Bayesian inference. In this work, we take a bottom up approach and ask: taking EFE as given, what's the resulting agent's optimality gap compared with a reward-driven reinforcement learning (RL) agent, which is well understood? By casting EFE under a particular class of belief MDP and using analysis tools from RL theory, we show that EFE approximates the Bayes optimal RL policy via information value. We discuss the implications for objective specification of active inference agents.

Related papers

• For Better or Worse: The Impact of Counterfactual Explanations' Directionality on User Behavior in xAI [6.883906273999368]
  Counterfactual explanations (CFEs) are a popular approach in explainable artificial intelligence (xAI) CFEs describe a scenario that is better than the factual state (upward CFE), or a scenario that is worse than the factual state (downward CFE) This study compares the impact of CFE directionality on behavior and experience of participants tasked to extract new knowledge from an automated system.
  arXiv  Detail & Related papers  (2023-06-13T09:16:38Z)
• Prediction-Oriented Bayesian Active Learning [51.426960808684655]
  Expected predictive information gain (EPIG) is an acquisition function that measures information gain in the space of predictions rather than parameters. EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models.
  arXiv  Detail & Related papers  (2023-04-17T10:59:57Z)
• Active Inference and Reinforcement Learning: A unified inference on continuous state and action spaces under partial observability [19.56438470022024]
  Many real-world problems involve partial observations, formulated as partially observable decision processes (POMDPs) Previous studies have tackled RL in POMDPs by either incorporating the memory of past actions and observations or by inferring the true state of the environment. We propose a unified principle that establishes a theoretical connection between Active inference (AIF) andReinforcement learning (RL) Experimental results demonstrate the superior learning capabilities of our method in solving continuous space partially observable tasks.
  arXiv  Detail & Related papers  (2022-12-15T16:28:06Z)
• Pseudo-Spherical Contrastive Divergence [119.28384561517292]
  We propose pseudo-spherical contrastive divergence (PS-CD) to generalize maximum learning likelihood of energy-based models. PS-CD avoids the intractable partition function and provides a generalized family of learning objectives.
  arXiv  Detail & Related papers  (2021-11-01T09:17:15Z)
• Active inference, Bayesian optimal design, and expected utility [1.433758865948252]
  We describe how active inference combines Bayesian decision theory and optimal Bayesian design principles to minimize expected free energy. It is this aspect of active inference that allows for the natural emergence of information-seeking behavior. Our Tmaze simulations show optimizing expected free energy produces goal-directed information-seeking behavior while optimizing expected utility induces purely exploitive behavior.
  arXiv  Detail & Related papers  (2021-09-21T20:56:32Z)
• Active Inference and Epistemic Value in Graphical Models [3.9457043990895904]
  The Free Energy Principle (FEP) postulates that biological agents perceive and interact with their environment in order to minimize a Variational Free Energy (VFE) with respect to a generative model of their environment. This paper approaches epistemic behavior from a constrained Bethe Free Energy (CBFE) perspective. We illustrate resulting behavior of the CBFE by planning and interacting with a simulated T-maze environment.
  arXiv  Detail & Related papers  (2021-09-01T16:43:35Z)
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation [85.75516599931632]
  We study the off-policy evaluation problem in reinforcement learning with linear function approximation. We propose an algorithm, VA-OPE, which uses the estimated variance of the value function to reweight the Bellman residual in Fitted Q-Iteration.
  arXiv  Detail & Related papers  (2021-06-22T17:58:46Z)
• Prior Preference Learning from Experts:Designing a Reward with Active Inference [1.1602089225841632]
  We claim that active inference can be interpreted using reinforcement learning (RL) algorithms. Motivated by the concept of prior preference and a theoretical connection, we propose a simple but novel method for learning a prior preference from experts.
  arXiv  Detail & Related papers  (2021-01-22T04:03:45Z)
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward [64.24528565312463]
  This paper tackles the challenge of using belief-based rewards for a deep RL agent. We derive the exact error between negative entropy and the expected prediction reward. This insight provides theoretical motivation for several fields using prediction rewards.
  arXiv  Detail & Related papers  (2020-05-11T08:13:49Z)
• Whence the Expected Free Energy? [68.8204255655161]
  We show that the Expected Free Energy (EFE) is not simply "the free energy in the future" We then develop a novel objective, the Free-Energy of the Expected Future (FEEF)
  arXiv  Detail & Related papers  (2020-04-17T09:06:56Z)
• Reinforcement Learning through Active Inference [62.997667081978825]
  We show how ideas from active inference can augment traditional reinforcement learning approaches. We develop and implement a novel objective for decision making, which we term the free energy of the expected future. We demonstrate that the resulting algorithm successfully exploration and exploitation, simultaneously achieving robust performance on several challenging RL benchmarks with sparse, well-shaped, and no rewards.
  arXiv  Detail & Related papers  (2020-02-28T10:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.