Boosting MCTS with Free Energy Minimization

• URL: http://arxiv.org/abs/2501.13083v1
• Date: Wed, 22 Jan 2025 18:45:15 GMT
• Title: Boosting MCTS with Free Energy Minimization
• Abstract summary: We propose a new planning framework that integrates Monte Carlo Tree Search (MCTS) with active inference objectives.<n>MCTS can be naturally extended to incorporate free energy minimization by blending expected rewards with information gain.<n>This synergy allows our planner to maintain coherent estimates of value and uncertainty throughout planning, without sacrificing computational tractability.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Active Inference, grounded in the Free Energy Principle, provides a powerful lens for understanding how agents balance exploration and goal-directed behavior in uncertain environments. Here, we propose a new planning framework, that integrates Monte Carlo Tree Search (MCTS) with active inference objectives to systematically reduce epistemic uncertainty while pursuing extrinsic rewards. Our key insight is that MCTS already renowned for its search efficiency can be naturally extended to incorporate free energy minimization by blending expected rewards with information gain. Concretely, the Cross-Entropy Method (CEM) is used to optimize action proposals at the root node, while tree expansions leverage reward modeling alongside intrinsic exploration bonuses. This synergy allows our planner to maintain coherent estimates of value and uncertainty throughout planning, without sacrificing computational tractability. Empirically, we benchmark our planner on a diverse set of continuous control tasks, where it demonstrates performance gains over both standalone CEM and MCTS with random rollouts.

Related papers

• Learning to Run Power Networks: Effective AlphaZero-inspired Topological Control [4.9058221615757]
  This paper investigates the effectiveness of model-based AlphaZero-inspired approaches for proactive grid management.<n>We show that the optimized AlphaZero approach achieves a peak survivability of 98.43%, significantly outperforming the proximal policy optimization (PPO) variant.<n>While AlphaZero is a powerful framework for topological control, pure reinforcement learning is not sufficient; rather, an effective and reliable system requires a'minimalist' integration of domain-specifics, binary rewards, and a restricted observation space of line loads.
  arXiv  Detail & Related papers  (2026-08-14T09:16:10Z)
• Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation [23.371552518874807]
  Automated Reward Design (ARD) aims to replace manual reward engineering in reinforcement learning with language-driven reward function synthesis.<n>Existing approaches based on large language models (LLMs) remain inherently correlation-driven, relying on iterative environmental feedback to refine reward hypotheses for each specific task.<n>We propose the Causal Reward World Model (CRWM), which explicitly models the causal relationships between candidate reward components and task-targeted physical variables through offline pre-training on multi-task interaction data.
  arXiv  Detail & Related papers  (2026-06-22T12:57:17Z)
• Expected Free Energy-based Planning as Variational Inference [1.2731778729620478]
  We show that EFE-based planning can be formulated as Variational Free Energy minimization.<n>Our main result demonstrates that minimizing a Variational Free Energy functional with appropriately chosen priors yields a decomposition into expected plan costs.
  arXiv  Detail & Related papers  (2026-06-09T08:09:48Z)
• Energy-Structured Low-Rank Adaptation for Continual Learning [78.5390284258307]
  We propose textbfEnergy-Concentrated and textbfEnergy-Ordered textbfLow-textbfRank textbfAdaptation (E$2$-LoRA)<n>By explicitly ordering and concentrating knowledge into leading ranks, E$2$-LoRA frees capacity for subsequent tasks.
  arXiv  Detail & Related papers  (2026-05-26T12:55:02Z)
• SGA-MCTS: Decoupling Planning from Execution via Training-Free Atomic Experience Retrieval [74.1918709002557]
  We introduce textbfSGA-MCTS, a framework that casts LLM planning as non-parametric retrieval.<n>Online, a retrieval-augmented agent employs a hybrid symbolic-semantic mechanism to fetch relevant State-Goal-Action atoms.<n>By effectively amortizing the heavy computational cost of search, SGA-MCTS achieves System 2 reasoning depth at System 1 inference speeds, rendering autonomous planning both scalable and real-time feasible.
  arXiv  Detail & Related papers  (2026-04-16T07:22:36Z)
• Discovering Process-Outcome Credit in Multi-Step LLM Reasoning [3.584086358722852]
  Reinforcement Learning (RL) serves as a potent paradigm for enhancing reasoning capabilities in Large Language Models (LLMs)<n>We propose a novel framework designed to provide continuous reward signals.<n>Our model exhibits superior out-of-distribution robustness, demonstrating promising zero-shot transfer capabilities to unseen and challenging reasoning tasks.
  arXiv  Detail & Related papers  (2026-02-01T05:44:09Z)
• Reinforcement Learning-Based Energy-Aware Coverage Path Planning for Precision Agriculture [6.218614094739671]
  This paper proposes an energy-aware CPP framework grounded in Soft Actor-Critic (SAC) reinforcement learning.<n>To enable robust and adaptive decision-making under energy limitations, the framework integrates Conal Neural Networks (CNNs) for spatial feature extraction.<n>A dedicated reward function is designed to jointly optimize coverage efficiency, energy consumption, and return-to-base constraints.
  arXiv  Detail & Related papers  (2026-01-23T02:33:14Z)
• An Explainable Equity-Aware P2P Energy Trading Framework for Socio-Economically Diverse Microgrid [0.0]
  This paper proposes a novel framework that integrates multi-objective mixed-integer linear programming (MILP), cooperative game theory, and a dynamic equity-adjustment mechanism driven by reinforcement learning (RL)<n>The framework demonstrates peak demand reductions of up to 72.6%, and significant cooperative gains.
  arXiv  Detail & Related papers  (2025-07-24T18:38:51Z)
• Expected Free Energy-based Planning as Variational Inference [3.559846046435839]
  We show that EFE-based planning arises naturally from minimizing a variational free energy functional on a generative model. This result reinforces theoretical consistency with the Free Energy Principle by casting planning under uncertainty itself as a form of variational inference. Our formulation yields policies that jointly support goal achievement and information gain, while incorporating a complexity term that accounts for bounded computational resources.
  arXiv  Detail & Related papers  (2025-04-21T07:09:05Z)
• Monte Carlo Tree Diffusion for System 2 Planning [57.50512800900167]
  We introduce Monte Carlo Tree Diffusion (MCTD), a novel framework that integrates the generative strength of diffusion models with the adaptive search capabilities of Monte Carlo Tree Search (MCTS) MCTD achieves the benefits of MCTS such as controlling exploration-exploitation trade-offs within the diffusion framework.
  arXiv  Detail & Related papers  (2025-02-11T02:51:42Z)
• Lipschitz Lifelong Monte Carlo Tree Search for Mastering Non-Stationary Tasks [19.42056439537988]
  This paper presents LiZero for Lipschitz lifelong planning using Monte Carlo Tree Search (MCTS) We propose a novel concept of adaptive UCT (aUCT) to transfer knowledge from a source task to the exploration/exploitation of a new task. Experiment results show that LiZero significantly outperforms existing MCTS and lifelong learning baselines in terms of much faster convergence to optimal rewards.
  arXiv  Detail & Related papers  (2025-02-02T02:45:20Z)
• Generalizing in Net-Zero Microgrids: A Study with Federated PPO and TRPO [5.195669033269619]
  This work addresses the challenge of optimal energy management in microgrids through a collaborative and privacy-preserving framework.<n>We propose the FedTRPO methodology, which integrates Federated Learning (FL) and Trust Region Policy Optimization (TRPO) to manage distributed energy resources efficiently.
  arXiv  Detail & Related papers  (2024-12-30T13:38:31Z)
• Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning [21.995159117991278]
  We propose Curiosity CEM, an improved version of the Cross-Entropy Method (CEM) algorithm for encouraging exploration via curiosity. Our proposed method maximizes the sum of state-action Q values over the planning horizon, in which these Q values estimate the future extrinsic and intrinsic reward. Experiments on image-based continuous control tasks from the DeepMind Control suite show that CCEM is by a large margin more sample-efficient than previous MBRL algorithms.
  arXiv  Detail & Related papers  (2023-03-07T10:48:20Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• Mitigating Out-of-Distribution Data Density Overestimation in Energy-Based Models [54.06799491319278]
  Deep energy-based models (EBMs) are receiving increasing attention due to their ability to learn complex distributions. To train deep EBMs, the maximum likelihood estimation (MLE) with short-run Langevin Monte Carlo (LMC) is often used. We investigate why the MLE with short-run LMC can converge to EBMs with wrong density estimates.
  arXiv  Detail & Related papers  (2022-05-30T02:49:17Z)
• Exploiting Submodular Value Functions For Scaling Up Active Perception [60.81276437097671]
  In active perception tasks, agent aims to select sensory actions that reduce uncertainty about one or more hidden variables. Partially observable Markov decision processes (POMDPs) provide a natural model for such problems. As the number of sensors available to the agent grows, the computational cost of POMDP planning grows exponentially.
  arXiv  Detail & Related papers  (2020-09-21T09:11:36Z)
• Reinforcement Learning through Active Inference [62.997667081978825]
  We show how ideas from active inference can augment traditional reinforcement learning approaches. We develop and implement a novel objective for decision making, which we term the free energy of the expected future. We demonstrate that the resulting algorithm successfully exploration and exploitation, simultaneously achieving robust performance on several challenging RL benchmarks with sparse, well-shaped, and no rewards.
  arXiv  Detail & Related papers  (2020-02-28T10:28:21Z)
• Risk-Aware Energy Scheduling for Edge Computing with Microgrid: A Multi-Agent Deep Reinforcement Learning Approach [82.6692222294594]
  We study a risk-aware energy scheduling problem for a microgrid-powered MEC network. We derive the solution by applying a multi-agent deep reinforcement learning (MADRL)-based advantage actor-critic (A3C) algorithm with shared neural networks.
  arXiv  Detail & Related papers  (2020-02-21T02:14:38Z)
• Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons [66.43848057122311]
  Reward tweaking learns a surrogate reward function that induces optimal behavior on the original finite-horizon total reward task. We show that reward tweaking guides the agent towards better long-horizon returns although it plans for short horizons.
  arXiv  Detail & Related papers  (2020-02-09T09:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.