Bayesian Experimental Design via Score Matching

• URL: http://arxiv.org/abs/2607.08335v1
• Date: Thu, 09 Jul 2026 10:21:06 GMT
• Title: Bayesian Experimental Design via Score Matching
• Abstract summary: We show that the double intractability of the expected information gain can be isolated from the policy learning.<n>This turns the key multiplicative cost into an additive one and reduces the computational burden on the policy training itself.<n>In our experiments we train multiple competitive policies without inducing a multiplicative cost in likelihood evaluations.
• Score: 15.998867609597545
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Policy-based approaches to Bayesian experimental design (BED) allow the learning of deep policy networks that adaptively make intelligent design decisions based on previously collected data. However, the training of such policies is often held back by a fundamental challenge: the double intractability of the expected information gain (EIG). This necessitates expensive or complex approximations that restrict the effort one can invest in optimising the policy itself. To address this, we show that the double intractability of the EIG can be isolated from the policy learning by first solving a score matching problem that is independent of the policy used, then using the learned score approximation to train the policy in a singly intractable manner. This turns the key multiplicative cost into an additive one and reduces the computational burden on the policy training itself, making it far cheaper to train the policy multiple times when needed, e.g. for architecture search, hyperparameter tuning, or avoiding local optima. In our experiments we train multiple competitive policies without inducing a multiplicative cost in likelihood evaluations, which can increase performance by allowing us to select the best policy even without performing hyperparameter or architecture searches.

Related papers

• Exact Model-Free Policy Iteration for Co-safe LTL Planning [2.5735476569508995]
  This work studies model-free reinforcement learning for co-safe linear temporal logic objectives in finite Markov decision processes.<n>We develop a new two-step model-free reinforcement learning method that first uses a discounted surrogate to identify a clamp set.<n>We prove almost-sure convergence of the policy evaluation step and finite termination of the policy iteration algorithm at an optimal policy.
  arXiv  Detail & Related papers  (2026-08-05T16:58:25Z)
• Intrinsic Reward Policy Optimization for Sparse-Reward Environments [1.9336815376402718]
  Intrinsic rewards can provide principled guidance for exploration.<n>We propose a policy optimization framework that leverages multiple intrinsic rewards to directly optimize a policy.<n>Our algorithm -- intrinsic reward policy optimization (IRPO) -- achieves this by using a surrogate policy gradient.
  arXiv  Detail & Related papers  (2026-01-29T08:25:14Z)
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning [52.97053840476386]
  We show that well-designed behaviour policies can be used to collect off-policy data for provably lower variance return estimates.<n>We extend this key insight to the online reinforcement learning setting, where both policy evaluation and improvement are interleaved.
  arXiv  Detail & Related papers  (2025-11-13T23:06:40Z)
• EXPO: Stable Reinforcement Learning with Expressive Policies [74.30151915786233]
  We propose a sample-efficient online reinforcement learning algorithm to maximize value with two parameterized policies.<n>Our approach yields up to 2-3x improvement in sample efficiency on average over prior methods.
  arXiv  Detail & Related papers  (2025-07-10T17:57:46Z)
• Convergence and Sample Complexity of First-Order Methods for Agnostic Reinforcement Learning [66.4260157478436]
  We study reinforcement learning in the policy learning setting.<n>The goal is to find a policy whose performance is competitive with the best policy in a given class of interest.
  arXiv  Detail & Related papers  (2025-07-06T14:40:05Z)
• Quantile-Optimal Policy Learning under Unmeasured Confounding [55.72891849926314]
  We study quantile-optimal policy learning where the goal is to find a policy whose reward distribution has the largest $alpha$-quantile for some $alpha in (0, 1)$.<n>Such a problem suffers from three main challenges: (i) nonlinearity of the quantile objective as a functional of the reward distribution, (ii) unobserved confounding issue, and (iii) insufficient coverage of the offline dataset.
  arXiv  Detail & Related papers  (2025-06-08T13:37:38Z)
• TD-M(PC)$^2$: Improving Temporal Difference MPC Through Policy Constraint [11.347808936693152]
  Model-based reinforcement learning algorithms combine model-based planning and learned value/policy prior.<n>Existing methods that rely on standard SAC-style policy iteration for value learning often result in emphpersistent value overestimation.<n>We propose a policy regularization term reducing out-of-distribution (OOD) queries, thereby improving value learning.
  arXiv  Detail & Related papers  (2025-02-05T19:08:42Z)
• $Δ\text{-}{\rm OPE}$: Off-Policy Estimation with Pairs of Policies [13.528097424046823]
  We introduce $Deltatext-rm OPE$ methods based on the widely used Inverse Propensity Scoring estimator. Simulated, offline, and online experiments show that our methods significantly improve performance for both evaluation and learning tasks.
  arXiv  Detail & Related papers  (2024-05-16T12:04:55Z)
• IOB: Integrating Optimization Transfer and Behavior Transfer for Multi-Policy Reuse [50.90781542323258]
  Reinforcement learning (RL) agents can transfer knowledge from source policies to a related target task. Previous methods introduce additional components, such as hierarchical policies or estimations of source policies' value functions. We propose a novel transfer RL method that selects the source policy without training extra components.
  arXiv  Detail & Related papers  (2023-08-14T09:22:35Z)
• CUP: Critic-Guided Policy Reuse [37.12379523150601]
  Critic-gUided Policy reuse (CUP) is a policy reuse algorithm that avoids training any extra components and efficiently reuses source policies. CUP selects the source policy that has the largest one-step improvement over the current target policy, and forms a guidance policy. Empirical results demonstrate that CUP achieves efficient transfer and significantly outperforms baseline algorithms.
  arXiv  Detail & Related papers  (2022-10-15T00:53:03Z)
• Efficient Deep Reinforcement Learning via Adaptive Policy Transfer [50.51637231309424]
  Policy Transfer Framework (PTF) is proposed to accelerate Reinforcement Learning (RL) Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods.
  arXiv  Detail & Related papers  (2020-02-19T07:30:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.