Trajectory-Regularized Stochastic Optimal Control via KL Divergence

• URL: http://arxiv.org/abs/2607.22201v1
• Date: Fri, 24 Jul 2026 11:15:27 GMT
• Title: Trajectory-Regularized Stochastic Optimal Control via KL Divergence
• Abstract summary: We introduce trajectory-regularized optimal control (TRSOC), which augments standard optimal control (SOC)<n>We derive the corresponding Hamilton-Jacobi-Bellman equation and characterize the optimal policy.<n>Experiments show that the regularization parameter induces a trade-off between performance-driven and reference-preserving behavior.
• Score: 8.729531978655737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce trajectory-regularized stochastic optimal control (TRSOC), which augments standard stochastic optimal control (SOC) with a Kullback--Leibler (KL) divergence between controlled and reference trajectory distributions. Using Girsanov's theorem, the trajectory KL reduces to a quadratic drift mismatch penalty, yielding a modified running cost that preserves the dynamic programming (DP) structure. We derive the corresponding Hamilton--Jacobi--Bellman (HJB) equation and characterize the optimal policy. In the linear-quadratic (LQ) setting, the formulation admits a closed-form solution with an augmented control cost. Experiments show that the regularization parameter induces a trade-off between performance-driven and reference-preserving behavior, including cases with reference dynamics learned from offline data.

Related papers

• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• Language Generation as Optimal Control: Closed-Loop Diffusion in Latent Control Space [99.30250506439678]
  This work reformulates language generation as an optimal control problem.<n>It provides a unified theoretical perspective to analyze autoregressive and diffusion models.
  arXiv  Detail & Related papers  (2026-05-14T08:13:43Z)
• Conservative Continuous-Time Treatment Optimization [9.553261527696067]
  We develop a conservative continuous-time control framework for treatment optimization from irregularly sampled patient trajectories.<n>Experiments on benchmark datasets show improved robustness and performance compared to non-conservative baselines.
  arXiv  Detail & Related papers  (2026-03-17T17:01:23Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Terminally constrained flow-based generative models from an optimal control perspective [32.87833798690545]
  Terminal Optimal Control with Flow-based models (TOCFlow) is a geometry-aware sampling-time guidance method for pre-trained flows.<n>We show that as the control penalty increases, the controlled process recovers the reference distribution, while as the penalty vanishes, the terminal law converges to a generalized Wasserstein projection onto the constraint manifold.<n>We evaluate TOCFlow on three high-dimensional scientific tasks spanning equality, inequality, and global statistical constraints.
  arXiv  Detail & Related papers  (2026-01-14T13:32:15Z)
• Unifying Entropy Regularization in Optimal Control: From and Back to Classical Objectives via Iterated Soft Policies and Path Integral Solutions [4.934817254755008]
  This paper develops a unified perspective on several optimal control formulations through the lens of Kullback-Leibler regularization.<n>We propose a central problem that separates the KL penalties on policies and transitions, assigning them independent weights.<n>We show that these soft-policy formulations majorize the original SOC and RSOC problem. This means that the regularized solution can be iterated to retrieve the original solution.
  arXiv  Detail & Related papers  (2025-12-05T19:31:39Z)
• Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning [56.47948583452555]
  We introduce the Stepwise Flow Policy (SWFP) framework, founded on the key insight that discretizing the flow matching inference process via a fixed-step Euler scheme aligns it with the variational Jordan-Kinderlehrer-Otto principle from optimal transport.<n>SWFP decomposes the global flow into a sequence of small, incremental transformations between proximate distributions.<n>This decomposition yields an efficient algorithm that fine-tunes pre-trained flows via a cascade of small flow blocks, offering significant advantages.
  arXiv  Detail & Related papers  (2025-10-17T07:43:51Z)
• Stochastic Optimal Control Matching [53.156277491861985]
  Our work introduces Optimal Control Matching (SOCM), a novel Iterative Diffusion Optimization (IDO) technique for optimal control. The control is learned via a least squares problem by trying to fit a matching vector field. Experimentally, our algorithm achieves lower error than all the existing IDO techniques for optimal control.
  arXiv  Detail & Related papers  (2023-12-04T16:49:43Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Adaptive Control and Regret Minimization in Linear Quadratic Gaussian (LQG) Setting [91.43582419264763]
  We propose LqgOpt, a novel reinforcement learning algorithm based on the principle of optimism in the face of uncertainty. LqgOpt efficiently explores the system dynamics, estimates the model parameters up to their confidence interval, and deploys the controller of the most optimistic model.
  arXiv  Detail & Related papers  (2020-03-12T19:56:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.