Sophisticated Inference

• URL: http://arxiv.org/abs/2006.04120v1
• Date: Sun, 7 Jun 2020 11:18:58 GMT
• Title: Sophisticated Inference
• Authors: Karl Friston, Lancelot Da Costa, Danijar Hafner, Casper Hesp, Thomas Parr
• Abstract summary: Active inference offers a first principle account of sentient behaviour. It replaces value functions with functionals of (Bayesian) beliefs. In this paper, we consider a sophisticated kind of active inference.
• Score: 8.145323363883234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active inference offers a first principle account of sentient behaviour, from which special and important cases can be derived, e.g., reinforcement learning, active learning, Bayes optimal inference, Bayes optimal design, etc. Active inference resolves the exploitation-exploration dilemma in relation to prior preferences, by placing information gain on the same footing as reward or value. In brief, active inference replaces value functions with functionals of (Bayesian) beliefs, in the form of an expected (variational) free energy. In this paper, we consider a sophisticated kind of active inference, using a recursive form of expected free energy. Sophistication describes the degree to which an agent has beliefs about beliefs. We consider agents with beliefs about the counterfactual consequences of action for states of affairs and beliefs about those latent states. In other words, we move from simply considering beliefs about 'what would happen if I did that' to 'what would I believe about what would happen if I did that'. The recursive form of the free energy functional effectively implements a deep tree search over actions and outcomes in the future. Crucially, this search is over sequences of belief states, as opposed to states per se. We illustrate the competence of this scheme, using numerical simulations of deep decision problems.

Related papers

• Reframing the Expected Free Energy: Four Formulations and a Unification [3.9121134770873733]
  Active inference is based on the expected free energy. This paper seek to formalize the problem of deriving these formulations from a single root expected free energy definition.
  arXiv  Detail & Related papers  (2024-02-22T11:38:43Z)
• Observing Interventions: A logic for thinking about experiments [62.997667081978825]
  This paper makes a first step towards a logic of learning from experiments. Crucial for our approach is the idea that the notion of an intervention can be used as a formal expression of a (real or hypothetical) experiment. For all the proposed logical systems, we provide a sound and complete axiomatization.
  arXiv  Detail & Related papers  (2021-11-25T09:26:45Z)
• Active inference, Bayesian optimal design, and expected utility [1.433758865948252]
  We describe how active inference combines Bayesian decision theory and optimal Bayesian design principles to minimize expected free energy. It is this aspect of active inference that allows for the natural emergence of information-seeking behavior. Our Tmaze simulations show optimizing expected free energy produces goal-directed information-seeking behavior while optimizing expected utility induces purely exploitive behavior.
  arXiv  Detail & Related papers  (2021-09-21T20:56:32Z)
• Nested Counterfactual Identification from Arbitrary Surrogate Experiments [95.48089725859298]
  We study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. Specifically, we prove the counterfactual unnesting theorem (CUT), which allows one to map arbitrary nested counterfactuals to unnested ones.
  arXiv  Detail & Related papers  (2021-07-07T12:51:04Z)
• Prior Preference Learning from Experts:Designing a Reward with Active Inference [1.1602089225841632]
  We claim that active inference can be interpreted using reinforcement learning (RL) algorithms. Motivated by the concept of prior preference and a theoretical connection, we propose a simple but novel method for learning a prior preference from experts.
  arXiv  Detail & Related papers  (2021-01-22T04:03:45Z)
• Counterfactual Credit Assignment in Model-Free Reinforcement Learning [47.79277857377155]
  Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards. We adapt the notion of counterfactuals from causality theory to a model-free RL setup. We formulate a family of policy algorithms that use future-conditional value functions as baselines or critics, and show that they are provably low variance.
  arXiv  Detail & Related papers  (2020-11-18T18:41:44Z)
• Learning "What-if" Explanations for Sequential Decision-Making [92.8311073739295]
  Building interpretable parameterizations of real-world decision-making on the basis of demonstrated behavior is essential. We propose learning explanations of expert decisions by modeling their reward function in terms of preferences with respect to "what if" outcomes. We highlight the effectiveness of our batch, counterfactual inverse reinforcement learning approach in recovering accurate and interpretable descriptions of behavior.
  arXiv  Detail & Related papers  (2020-07-02T14:24:17Z)
• Maximizing Information Gain in Partially Observable Environments via Prediction Reward [64.24528565312463]
  This paper tackles the challenge of using belief-based rewards for a deep RL agent. We derive the exact error between negative entropy and the expected prediction reward. This insight provides theoretical motivation for several fields using prediction rewards.
  arXiv  Detail & Related papers  (2020-05-11T08:13:49Z)
• Whence the Expected Free Energy? [68.8204255655161]
  We show that the Expected Free Energy (EFE) is not simply "the free energy in the future" We then develop a novel objective, the Free-Energy of the Expected Future (FEEF)
  arXiv  Detail & Related papers  (2020-04-17T09:06:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.