Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models

• URL: http://arxiv.org/abs/2605.07701v1
• Date: Fri, 08 May 2026 13:12:32 GMT
• Title: Guidance Is Not a Hyperparameter: Learning Dynamic Control in Diffusion Language Models
• Abstract summary: We recast CFG scale selection as a sequential decision-making problem and propose to learn dynamic guidance trajectories via reinforcement learning.<n>Experiments on three controlled NLP generation tasks using discrete diffusion language models demonstrate that adaptive guidance consistently achieves a better balance between controllability and generation quality than fixed-scale strategies.
• Score: 8.501624260424387
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classifier-Free Guidance (CFG) is a widely used mechanism for controlling diffusion-based generative models, yet its guidance scale is typically treated as a fixed hyperparameter throughout generation. This static design yields a suboptimal controllability and quality tradeoff, as the optimal degree of guidance varies across tasks and across different stages of the diffusion process, especially in NLP domain. We recast CFG scale selection as a sequential decision-making problem and propose to learn dynamic guidance trajectories via reinforcement learning. Specifically, we model the guidance scale as a discrete control action selected at each generation step based on the evolving diffusion state, and optimize a policy using Proximal Policy Optimization (PPO) under task-level rewards. Experiments on three controlled NLP generation tasks using discrete diffusion language models demonstrate that adaptive guidance consistently achieves a better balance between controllability and generation quality than fixed-scale strategies. Further analysis of the learned policies reveals distinct and interpretable guidance trajectories across tasks, underscoring the importance of treating guidance as a dynamic control process rather than a static design choice.

Related papers

• From Noise to Control: Parameterized Diffusion Policies [54.941343992110426]
  We propose a framework for learning diffusion policies conditioned on continuous parameters embedded in a learned behavior manifold.<n>Our approach enables smooth distances between known strategies and efficient adaptation to novel constraints without updating policy weights.
  arXiv  Detail & Related papers  (2026-05-29T20:21:50Z)
• Adaptive Linear Path Model-Based Diffusion [52.84663832658799]
  We introduce Linear Path Model-Based Diffusion (LP-MBD), which replaces the variance-preserving schedule with a flow-matching linear probability path.<n>We also propose Adaptive LP-MBD (ALP-MBD), which leverages reinforcement learning to adjust diffusion steps and noise levels according to task complexity and environmental conditions.
  arXiv  Detail & Related papers  (2026-02-02T21:33:03Z)
• Autonomous Reasoning for Spacecraft Control: A Large Language Model Framework with Group Relative Policy Optimization [2.258690092379457]
  This paper presents a learning-based guidance-and-control approach that couples a reasoning-enabled Large Language Model (LLM) with Group Relative Policy Optimization ( GRPO)<n>A two-stage procedure consisting of Supervised Fine-Tuning (SFT) to learn formatting and control primitives, followed by GRPO for interaction-driven policy improvement, trains controllers for each environment.<n>Results demonstrate that an LLM with explicit reasoning, optimized via GRPO, can synthesize feasible stabilizing policies under consistent training settings across both linear and nonlinear systems.
  arXiv  Detail & Related papers  (2026-01-07T19:13:22Z)
• Model-Based Diffusion Sampling for Predictive Control in Offline Decision Making [48.998030470623384]
  offline decision-making requires reliable behaviors from fixed datasets without further interaction.<n>We propose a compositional model-based diffusion framework consisting of: (i) a planner that generates diverse, task-aligned trajectories; (ii) a dynamics model that enforces consistency with the underlying system dynamics; and (iii) a ranker module that selects behaviors aligned with the task objectives.
  arXiv  Detail & Related papers  (2025-12-09T06:26:02Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Drive As You Like: Strategy-Level Motion Planning Based on A Multi-Head Diffusion Model [7.3078271605135114]
  We propose a diffusion-based multi-head trajectory planner(M-diffusion planner)<n>During the early training stage, all output heads share weights to learn to generate high-quality trajectories.<n>We incorporate a large language model (LLM) to guide strategy selection, enabling dynamic, instruction-aware planning.
  arXiv  Detail & Related papers  (2025-08-23T08:33:11Z)
• Prior-Guided Diffusion Planning for Offline Reinforcement Learning [5.819784482811376]
  Prior Guidance (PG) is a novel guided sampling framework that replaces the standard Gaussian prior of a behavior-cloned diffusion model.<n>We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-the-art diffusion policies and planners across diverse long-horizon offline RL benchmarks.
  arXiv  Detail & Related papers  (2025-05-16T05:39:02Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control [46.81433026280051]
  We present a self-supervised learning approach that actively models the dynamics of nonlinear robotic systems. Our approach showcases high resilience and generalization capabilities by consistently adapting to unseen flight conditions.
  arXiv  Detail & Related papers  (2022-10-23T00:45:05Z)
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning [137.39196753245105]
  We present a new model-based reinforcement learning algorithm that learns a multi-headed dynamics model for dynamics generalization. We incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods.
  arXiv  Detail & Related papers  (2020-10-26T03:20:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.