Prior preferences in active inference agents: soft, hard, and goal shaping

• URL: http://arxiv.org/abs/2512.03293v1
• Date: Tue, 02 Dec 2025 23:07:24 GMT
• Title: Prior preferences in active inference agents: soft, hard, and goal shaping
• Authors: Filippo Torresan, Ryota Kanai, Manuel Baltieri,
• Abstract summary: Active inference proposes expected free energy as an objective to balance exploitative and explorative drives in learning agents.<n>We consider four possible ways of defining the preference distribution, either providing the agents with hard or soft goals.<n>We show that goal shaping enables the best performance overall (i.e., it promotes exploitation) while sacrificing learning about the environment's transition dynamics.
• Score: 3.2776596620344285
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Active inference proposes expected free energy as an objective for planning and decision-making to adequately balance exploitative and explorative drives in learning agents. The exploitative drive, or what an agent wants to achieve, is formalised as the Kullback-Leibler divergence between a variational probability distribution, updated at each inference step, and a preference probability distribution that indicates what states or observations are more likely for the agent, hence determining the agent's goal in a certain environment. In the literature, the questions of how the preference distribution should be specified and of how a certain specification impacts inference and learning in an active inference agent have been given hardly any attention. In this work, we consider four possible ways of defining the preference distribution, either providing the agents with hard or soft goals and either involving or not goal shaping (i.e., intermediate goals). We compare the performances of four agents, each given one of the possible preference distributions, in a grid world navigation task. Our results show that goal shaping enables the best performance overall (i.e., it promotes exploitation) while sacrificing learning about the environment's transition dynamics (i.e., it hampers exploration).

Related papers

• Desirable Effort Fairness and Optimality Trade-offs in Strategic Learning [4.702729080310267]
  We study how decision rules interact with agents who may strategically change their inputs/features to achieve better outcomes.<n>We propose a unified model of principal-agent interaction that captures this trade-off.
  arXiv  Detail & Related papers  (2025-10-21T21:43:20Z)
• Goal-Directedness is in the Eye of the Beholder [48.937781898861815]
  Probing for goal-directed behavior comes in two flavors: Behavioral and mechanistic.<n>We identify technical and conceptual problems that arise from formalizing goals in agent systems.<n>We outline new directions for modeling goal-directedness as an emergent property of dynamic, multi-agent systems.
  arXiv  Detail & Related papers  (2025-08-18T11:04:18Z)
• Performative Prediction on Games and Mechanism Design [69.7933059664256]
  We study a collective risk dilemma where agents decide whether to trust predictions based on past accuracy.<n>As predictions shape collective outcomes, social welfare arises naturally as a metric of concern.<n>We show how to achieve better trade-offs and use them for mechanism design.
  arXiv  Detail & Related papers  (2024-08-09T16:03:44Z)
• Statistical Inference Under Constrained Selection Bias [20.862583584531322]
  We propose a framework that enables statistical inference in the presence of selection bias. The output is high-probability bounds on the value of an estimand for the target distribution. We analyze the computational and statistical properties of methods to estimate these bounds and show that our method can produce informative bounds on a variety of simulated and semisynthetic tasks.
  arXiv  Detail & Related papers  (2023-06-05T23:05:26Z)
• Control-Aware Prediction Objectives for Autonomous Driving [78.19515972466063]
  We present control-aware prediction objectives (CAPOs) to evaluate the downstream effect of predictions on control without requiring the planner be differentiable. We propose two types of importance weights that weight the predictive likelihood: one using an attention model between agents, and another based on control variation when exchanging predicted trajectories for ground truth trajectories.
  arXiv  Detail & Related papers  (2022-04-28T07:37:21Z)
• Generative multitask learning mitigates target-causing confounding [61.21582323566118]
  We propose a simple and scalable approach to causal representation learning for multitask learning. The improvement comes from mitigating unobserved confounders that cause the targets, but not the input. Our results on the Attributes of People and Taskonomy datasets reflect the conceptual improvement in robustness to prior probability shift.
  arXiv  Detail & Related papers  (2022-02-08T20:42:14Z)
• Explaining Reinforcement Learning Policies through Counterfactual Trajectories [147.7246109100945]
  A human developer must validate that an RL agent will perform well at test-time. Our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.
  arXiv  Detail & Related papers  (2022-01-29T00:52:37Z)
• Deceptive Decision-Making Under Uncertainty [25.197098169762356]
  We study the design of autonomous agents that are capable of deceiving outside observers about their intentions while carrying out tasks. By modeling the agent's behavior as a Markov decision process, we consider a setting where the agent aims to reach one of multiple potential goals. We propose a novel approach to model observer predictions based on the principle of maximum entropy and to efficiently generate deceptive strategies.
  arXiv  Detail & Related papers  (2021-09-14T14:56:23Z)
• Understanding the origin of information-seeking exploration in probabilistic objectives for control [62.997667081978825]
  An exploration-exploitation trade-off is central to the description of adaptive behaviour. One approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' We show that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives.
  arXiv  Detail & Related papers  (2021-03-11T18:42:39Z)
• Modulation of viability signals for self-regulatory control [1.370633147306388]
  We revisit the role of instrumental value as a driver of adaptive behavior. For reinforcement learning tasks, the distribution of preferences replaces the notion of reward.
  arXiv  Detail & Related papers  (2020-07-18T01:11:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.