Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics

• URL: http://arxiv.org/abs/2604.21456v1
• Date: Thu, 23 Apr 2026 09:13:59 GMT
• Title: Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics
• Abstract summary: We propose a sampling-based framework for finite-horizon trajectory and policy optimization under differentiable dynamics.<n>For policy optimization, we extend TSMC via (i) a deterministic empirical approximation of the initial-state distribution and (ii) an extended-space construction that treats rollout randomness as auxiliary variables.<n>Experiments across trajectory- and policy-optimization benchmarks show that TSMC is broadly applicable and compares favorably to state-of-the-art baselines.
• Score: 5.415983152637643
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a sampling-based framework for finite-horizon trajectory and policy optimization under differentiable dynamics by casting controller design as inference. Specifically, we minimize a KL-regularized expected trajectory cost, which yields an optimal "Boltzmann-tilted" distribution over controller parameters that concentrates on low-cost solutions as temperature decreases. To sample efficiently from this sharp, potentially multimodal target, we introduce tempered sequential Monte Carlo (TSMC): an annealing scheme that adaptively reweights and resamples particles along a tempering path from a prior to the target distribution, while using Hamiltonian Monte Carlo rejuvenation to maintain diversity and exploit exact gradients obtained by differentiating through trajectory rollouts. For policy optimization, we extend TSMC via (i) a deterministic empirical approximation of the initial-state distribution and (ii) an extended-space construction that treats rollout randomness as auxiliary variables. Experiments across trajectory- and policy-optimization benchmarks show that TSMC is broadly applicable and compares favorably to state-of-the-art baselines.

Related papers

• From Noise to Control: Parameterized Diffusion Policies [54.941343992110426]
  We propose a framework for learning diffusion policies conditioned on continuous parameters embedded in a learned behavior manifold.<n>Our approach enables smooth distances between known strategies and efficient adaptation to novel constraints without updating policy weights.
  arXiv  Detail & Related papers  (2026-05-29T20:21:50Z)
• Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo [15.105260031413698]
  We study inference-time alignment for diffusion-based generative models.<n>We propose a trust-region framework for learning twisting functions in SMC-based inference-time alignment.
  arXiv  Detail & Related papers  (2026-05-24T15:08:52Z)
• Flow Matching Policy with Entropy Regularization [16.47598359293598]
  Flow Matching Policy with Entropy Regularization (FMER) is an Ordinary Differential Equation (ODE)-based online RL framework.<n>FMER parameterizes the policy via flow matching and samples actions along a straight probability path, motivated by optimal transport.<n> Experiments on sparse multi-goal FrankaKitchen benchmarks demonstrate that FMER outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-03-18T13:00:20Z)
• Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser [35.260598916253635]
  We introduce Variance Minimisation Policy optimisation (VMPO)<n>VMPO formulates diffusion alignment as minimising the variance of log importance weights.<n>Under different choices of potential functions and variance minimisation strategies, VMPO recovers various existing methods.
  arXiv  Detail & Related papers  (2026-02-12T18:06:03Z)
• Reinforced sequential Monte Carlo for amortised sampling [49.92678178064033]
  We state a connection between sequential Monte Carlo (SMC) and neural sequential samplers trained by maximum-entropy reinforcement learning (MaxEnt RL)<n>We describe techniques for stable joint training of proposals and twist functions and an adaptive weight tempering scheme to reduce training signal variance.
  arXiv  Detail & Related papers  (2025-10-13T17:59:11Z)
• Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Design [17.7006862812979]
  We propose a Sequential Monte Carlo framework that enables scalable inference-time control of discrete diffusion models.<n>Specifically, our approach derives tractable importance weights for a range of intermediate targets and characterises the optimal proposal.<n> Empirical results across synthetic tasks, language modelling, biology design, and text-to-image generation demonstrate that our framework enhances controllability and sample quality.
  arXiv  Detail & Related papers  (2025-05-28T16:12:03Z)
• Policy Gradients for Optimal Parallel Tempering MCMC [0.276240219662896]
  Parallel tempering is a meta-algorithm for Markov Chain Monte Carlo that uses multiple chains to sample from tempered versions of the target distribution.<n>We present an adaptive temperature selection algorithm that dynamically adjusts temperatures during sampling using a policy gradient approach.
  arXiv  Detail & Related papers  (2024-09-03T03:12:45Z)
• Monte Carlo Policy Gradient Method for Binary Optimization [3.742634130733923]
  We develop a novel probabilistic model to sample the binary solution according to a parameterized policy distribution. For coherent exploration in discrete spaces, parallel Markov Chain Monte Carlo (MCMC) methods are employed. Convergence to stationary points in expectation of the policy gradient method is established.
  arXiv  Detail & Related papers  (2023-07-03T07:01:42Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• Differentiating Metropolis-Hastings to Optimize Intractable Densities [51.16801956665228]
  We develop an algorithm for automatic differentiation of Metropolis-Hastings samplers. We apply gradient-based optimization to objectives expressed as expectations over intractable target densities.
  arXiv  Detail & Related papers  (2023-06-13T17:56:02Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Comparing Probability Distributions with Conditional Transport [63.11403041984197]
  We propose conditional transport (CT) as a new divergence and approximate it with the amortized CT (ACT) cost. ACT amortizes the computation of its conditional transport plans and comes with unbiased sample gradients that are straightforward to compute. On a wide variety of benchmark datasets generative modeling, substituting the default statistical distance of an existing generative adversarial network with ACT is shown to consistently improve the performance.
  arXiv  Detail & Related papers  (2020-12-28T05:14:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.