Active Inference as a Convex Markov Decision Process

• URL: http://arxiv.org/abs/2607.20152v1
• Date: Wed, 22 Jul 2026 13:52:40 GMT
• Title: Active Inference as a Convex Markov Decision Process
• Abstract summary: Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE)<n>We show that, for closed-loop control policies, EFE minimization can be formulated as a Markov decision process (MDP)<n>We argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• Online Policy Evaluation for MDPs with Dynamic UBSR Measures [5.271072635937926]
  We propose efficient online learning algorithms for policy evaluation in Markov decision processes.<n>We introduce the UBSR-TD algorithm, establish conditions under which it converges almost surely, and develop several variants designed to accelerate convergence.<n>Our formulation shows that existing policy evaluation algorithms for risk-neutral MDPs can be readily adapted to dynamic UBSR settings.
  arXiv  Detail & Related papers  (2026-07-25T04:12:05Z)
• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models [9.133661763917374]
  We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces.<n>We develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models.
  arXiv  Detail & Related papers  (2026-07-16T03:19:53Z)
• Bounded Ratio Reinforcement Learning [62.09379476369526]
  This paper introduces the Bounded Ratio Reinforcement Learning (BRRL) framework to solve on-policy reinforcement learning problems.<n>We develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL.<n> Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM).
  arXiv  Detail & Related papers  (2026-04-20T17:59:01Z)
• Provable and Practical In-Context Policy Optimization for Self-Improvement [49.670847804409874]
  We study test-time scaling, where a model improves its answer through multi-round self-reflection at inference.<n>We introduce In-Context Policy Optimization (ICPO), in which an agent optimize its response in context using self-assessed or externally observed rewards without modifying its parameters.<n>We propose Minimum-Entropy ICPO (ME-ICPO), a practical algorithm that iteratively uses its response and self-assessed reward to refine its response in-context at inference time.
  arXiv  Detail & Related papers  (2026-03-02T00:21:50Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Policy Regularized Distributionally Robust Markov Decision Processes with Linear Function Approximation [10.35045003737115]
  Decision-making under distribution shift is a central challenge in reinforcement learning (RL), where training and deployment environments differ.<n>We propose DR-RPO, a model-free online policy optimization method that learns robust policies with sublinear regret.<n>We show that DR-RPO can achieve suboptimality bounds and sample efficiency in robust RL, matching the performance of value-based approaches.
  arXiv  Detail & Related papers  (2025-10-16T02:56:58Z)
• Sequential Monte Carlo for Policy Optimization in Continuous POMDPs [10.675160254513615]
  We introduce a novel policy optimization framework for continuous partially observable Markov decision processes.<n>Our method casts policy learning as probabilistic inference in a non-Markovian Feynman--Kac model.<n>We demonstrate the effectiveness of our algorithm across standard continuous POMDP benchmarks.
  arXiv  Detail & Related papers  (2025-05-22T14:45:46Z)
• Dual Alignment Maximin Optimization for Offline Model-based RL [37.563771012143725]
  offline reinforcement agents face significant deployment challenges due to the synthetic-to-real distribution mismatch.<n>In this paper, we first shift the focus from model reliability to policy discrepancies while optimizing for expected returns, and then self-consistently incorporate synthetic data.<n>It is a unified framework to ensure both model-environment policy consistency and synthetic and data offline.
  arXiv  Detail & Related papers  (2025-02-02T16:47:35Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• An Offline Risk-aware Policy Selection Method for Bayesian Markov Decision Processes [0.0]
  Exploitation vs Caution (EvC) is a paradigm that elegantly incorporates model uncertainty abiding by the Bayesian formalism. We validate EvC with state-of-the-art approaches in different discrete, yet simple, environments offering a fair variety of MDP classes. In the tested scenarios EvC manages to select robust policies and hence stands out as a useful tool for practitioners.
  arXiv  Detail & Related papers  (2021-05-27T20:12:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.