DiPRL: Learning Discrete Programmatic Policies via Architecture Entropy Regularization

• URL: http://arxiv.org/abs/2605.18508v1
• Date: Mon, 18 May 2026 15:01:15 GMT
• Title: DiPRL: Learning Discrete Programmatic Policies via Architecture Entropy Regularization
• Abstract summary: Programmatic reinforcement learning (PRL) offers an interpretable alternative to deep reinforcement learning.<n> gradient-based methods have been developed to optimize continuous relaxations of programs.<n>Post-hoc discretization can discard optimized branches and parameters in a program, which results in a collapse of policy expressivity.<n>We propose Differentiable Discrete Programmatic Reinforcement Learning (DiPRL) to overcome these limitations.
• Score: 3.903101278501413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Programmatic reinforcement learning (PRL) offers an interpretable alternative to deep reinforcement learning by representing policies as human-readable and -editable programs. While gradient-based methods have been developed to optimize continuous relaxations of programs, they face a significant performance drop when converting the continuous relaxations back into discrete programs. Post-hoc discretization can discard optimized branches and parameters in a program, which results in a collapse of policy expressivity and lowered task performance, leading in turn to a need for additional fine-tuning. To overcome these limitations, we propose Differentiable Discrete Programmatic Reinforcement Learning (DiPRL), a method that learns programmatic policies that become nearly discrete during training, avoiding a separate post-hoc fine-tuning stage. We first analyze the inherent risks of performance drop introduced by post-hoc discretization of gradient-based methods. Then, we introduce programmatic architecture entropy regularization, which enables smooth, differentiable training that encourages convergence toward a discrete program. DiPRL maintains the efficiency of gradient-based optimization while mitigating the risks of post-hoc discretization. Our experiments across multiple discrete and continuous RL tasks demonstrate that DiPRL can achieve strong performance via interpretable programmatic policies.

Related papers

• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models [9.133661763917374]
  We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces.<n>We develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models.
  arXiv  Detail & Related papers  (2026-07-16T03:19:53Z)
• Actor-Critic with Active Importance Sampling [61.41272490044518]
  This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm.<n>AISAC is an extension of the Actor-Critic framework for reducing variance in policy gradient estimation.<n> Experiments show improved learning speed, sample efficiency, and training compared to standard Actor-Critic methods.
  arXiv  Detail & Related papers  (2026-05-08T01:21:32Z)
• Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages [7.728607011076579]
  Reinforcement learning has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods.<n>We formulate a diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive an exact, unbiased policy gradient.<n>Experiments on coding and logical reasoning benchmarks demonstrate state-of-the-art results, with strong competitive performance on mathematical reasoning.
  arXiv  Detail & Related papers  (2026-03-13T01:38:44Z)
• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Optimistic Planning by Regularized Dynamic Programming [12.411844611718958]
  We propose a new method for optimistic planning in infinite-horizon discounted Markov decision processes. This technique allows us to avoid contraction and monotonicity arguments. We show it achieves near-optimal statistical guarantees.
  arXiv  Detail & Related papers  (2023-02-27T17:48:08Z)
• Hierarchical Programmatic Reinforcement Learning via Learning to Compose Programs [58.94569213396991]
  We propose a hierarchical programmatic reinforcement learning framework to produce program policies. By learning to compose programs, our proposed framework can produce program policies that describe out-of-distributionally complex behaviors. The experimental results in the Karel domain show that our proposed framework outperforms baselines.
  arXiv  Detail & Related papers  (2023-01-30T14:50:46Z)
• Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees [56.848265937921354]
  Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy. Many algorithms for IRL have an inherently nested structure. We develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy.
  arXiv  Detail & Related papers  (2022-10-04T17:13:45Z)
• Stochastic convex optimization for provably efficient apprenticeship learning [1.0609815608017066]
  We consider large-scale Markov decision processes (MDPs) with an unknown cost function. We employ convex optimization tools to address the problem of imitation learning, which consists of learning a policy from a finite set of expert demonstrations.
  arXiv  Detail & Related papers  (2021-12-31T19:47:57Z)
• Improper Learning with Gradient-based Policy Optimization [62.50997487685586]
  We consider an improper reinforcement learning setting where the learner is given M base controllers for an unknown Markov Decision Process. We propose a gradient-based approach that operates over a class of improper mixtures of the controllers.
  arXiv  Detail & Related papers  (2021-02-16T14:53:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.