Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators

• URL: http://arxiv.org/abs/2607.16858v1
• Date: Sat, 18 Jul 2026 15:44:45 GMT
• Title: Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators
• Abstract summary: unsupervised reinforcement learning requires intrinsic motivation that does not precommit to a particular direction of surprise.<n>Prediction-error curiosity rewards total expected surprise, including irreducible noise.<n>We propose a single intrinsic reward, stationary within each window, derived from the novelty contribution of a preference-free Expected Free Energy objective.
• Score: 7.280207127853575
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Across environments with mixed sources of uncertainty, unsupervised reinforcement learning requires intrinsic motivation that does not precommit to a particular direction of surprise. Surprise minimization is scoped by design to ``unstable'' environments. Prediction-error curiosity rewards total expected surprise, including irreducible noise. Bandit or mixture switching between surprise-minimizing and surprise-maximizing rewards reintroduces non-stationarity by construction. We propose a single intrinsic reward, stationary within each window, derived from the novelty contribution of a preference-free Expected Free Energy objective, expressed in reward-maximization form. Our claim is that parameter information gain, the expected surprise of the next state minus its irreducible part, is the appropriate intrinsic signal in both high-entropy and low-entropy components of the state space. Maximizing it seeks exactly the surprise the model can explain away. In regions of unresolved dynamics, this epistemic term drives exploration. As dynamics become resolved, the epistemic term vanishes, while an aleatoric penalty favors lower-variance transitions, all without fitting an explicit next-state predictor. A pseudocount supplies epistemic value, a probe-based penalty captures aleatoric variance, and a short-horizon gate protects informative successors. A window-based freeze of all reward-defining objects yields a stationary Bellman operator, explicit bounds on learning targets, and a conditional uniform-concentration result for the nonparametric estimators under mixing, smoothness, bandwidth, and capacity assumptions. In active-inference terms, the agent is preference-free where novelty is retained, standard likelihood ambiguity vanishes under full observability, a nonstandard transition-entropy penalty is added, and surprise minimization emerges in resolved regions of the state space.

Related papers

• Expected free energy as an information constraint on the Bethe Lagrangian [2.360534864805446]
  Active inference selects actions by minimising an expected free energy functional over predicted futures.<n>We propose an alternative formulation based on a Bethe free energy functional, fully supporting inference by message passing.<n>We show that, as the information demand is varied, the solved multiplier moves through its inactive, interior, and saturated regimes.
  arXiv  Detail & Related papers  (2026-08-17T22:10:54Z)
• High-dimensional ridgeless least squares interpolation under spiked covariance structures [3.656751299156262]
  This paper investigates the behavior of the out-of-sample prediction risk of the high-dimensional ridgeless least-squares estimator when the feature dimension $p$ and the sample size $n$ grow proportionally.<n>We show that the signal energy distributed along latent spike directions determines whether overfitting leads to benign, tempered, or catastrophic overfitting.
  arXiv  Detail & Related papers  (2026-08-07T14:40:44Z)
• Analytic Planning under Uncertainty with Moment Closure [32.403961716203455]
  Propagating full state analytically offers a principled way to do this, but has traditionally required restrictive policy or reward structures to remain tractable.<n>We investigate whether distribution-aware planning is possible without these constraints.<n>Our approach reduces target variance yields well-calibrated predictive uncertainty under observations in continuous control.
  arXiv  Detail & Related papers  (2026-08-03T17:17:00Z)
• Persistent Gaussian Perturbations Prevent Oversmoothing in Recurrent Graph Neural Networks [2.8427946758947304]
  We study a recurrent graph neural network in which independent Gaussian noise is injected after every propagation step.<n>We prove that the hidden representations form a geometrically ergodic Markov chain admitting a unique invariant probability measure.<n>Our analysis reveals persistent perturbations as a fundamentally different mechanism for combating oversmoothing.
  arXiv  Detail & Related papers  (2026-07-30T13:21:18Z)
• Predictable GRPO: A Closed-Form Model of Training Dynamics [1.0537323885285492]
  We develop a first-principles reduced-order model of these dynamics.<n>Group size enters, to leading order, as a noise temperature.<n>Across three models and two group sizes, the closed-form trajectory fits training reward to $R2 geq 0.91$.
  arXiv  Detail & Related papers  (2026-06-29T18:19:09Z)
• Gradient-Free Noise Optimization for Reward Alignment in Generative Models [52.42087778653147]
  ZeNO is a gradient-free framework that formulates noise optimization as a path-integral control problem.<n>It enables effective inference-time scaling and demonstrates strong performance across diverse generators and reward functions.
  arXiv  Detail & Related papers  (2026-05-12T00:05:36Z)
• CURVE: Learning Causality-Inspired Invariant Representations for Robust Scene Understanding via Uncertainty-Guided Regularization [30.613712415224473]
  CURVE is a framework that integrates variational uncertainty modeling with uncertainty-guided structural regularization to suppress high-variance relations.<n>Specifically, we apply prototype-conditioned debiasing to disentangle invariant interaction dynamics from environment-dependent variations, promoting a sparse and domain-stable topology.<n> Empirically, we evaluate CURVE in zero-shot transfer and low-data sim-to-real adaptation, verifying its ability to learn domain-stable sparse topologies and provide reliable uncertainty estimates to support risk prediction under distribution shifts.
  arXiv  Detail & Related papers  (2026-01-28T08:15:56Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• Equilibrium Propagation Without Limits [0.0]
  We prove that the gradient of the difference in Helmholtz free energy between a nudged and free phase is exactly the difference in expected local energy derivatives.<n>This validates the classic Contrastive Hebbian Learning update as an exact gradient estimator for arbitrary finite nudging.
  arXiv  Detail & Related papers  (2025-11-27T01:55:26Z)
• Exploring Semantic-constrained Adversarial Example with Instruction Uncertainty Reduction [51.50282796099369]
  This paper develops a multi-dimensional instruction uncertainty reduction framework to generate semantically constrained adversarial examples.<n>By predicting the language-guided sampling process, the optimization process will be stabilized by the designed ResAdv-DDIM sampler.<n>We realize the reference-free generation of semantically constrained 3D adversarial examples for the first time.
  arXiv  Detail & Related papers  (2025-10-27T04:02:52Z)
• Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression [17.49026509916207]
  Uncertainty quantification is critical for deploying deep neural networks (DNNs) in real-world applications. For vision regression tasks, current AuxUE designs are mainly adopted for aleatoric uncertainty estimates. We propose a generalized AuxUE scheme for more robust uncertainty quantification on regression tasks.
  arXiv  Detail & Related papers  (2023-08-17T15:54:11Z)
• Model-Based Uncertainty in Value Functions [89.31922008981735]
  We focus on characterizing the variance over values induced by a distribution over MDPs. Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation. We propose a new uncertainty Bellman equation whose solution converges to the true posterior variance over values.
  arXiv  Detail & Related papers  (2023-02-24T09:18:27Z)
• Neuro-Symbolic Entropy Regularization [78.16196949641079]
  In structured prediction, the goal is to jointly predict many output variables that together encode a structured object. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.
  arXiv  Detail & Related papers  (2022-01-25T06:23:10Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.