Physics-Informed Reward Machines

• URL: http://arxiv.org/abs/2508.14093v1
• Date: Thu, 14 Aug 2025 18:46:54 GMT
• Title: Physics-Informed Reward Machines
• Authors: Daniel Ajeleye, Ashutosh Trivedi, Majid Zamani,
• Abstract summary: Reward machines (RMs) provide a structured way to specify non-Markovian rewards in reinforcement learning (RL)<n>We introduce physics-informed reward machines (pRMs), a symbolic machine designed to express complex learning objectives and reward structures for RL agents.<n>We present RL algorithms capable of exploiting pRMs via counterfactual experiences and reward shaping.
• Score: 4.7962647777554634
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reward machines (RMs) provide a structured way to specify non-Markovian rewards in reinforcement learning (RL), thereby improving both expressiveness and programmability. Viewed more broadly, they separate what is known about the environment, captured by the reward mechanism, from what remains unknown and must be discovered through sampling. This separation supports techniques such as counterfactual experience generation and reward shaping, which reduce sample complexity and speed up learning. We introduce physics-informed reward machines (pRMs), a symbolic machine designed to express complex learning objectives and reward structures for RL agents, thereby enabling more programmable, expressive, and efficient learning. We present RL algorithms capable of exploiting pRMs via counterfactual experiences and reward shaping. Our experimental results show that these techniques accelerate reward acquisition during the training phases of RL. We demonstrate the expressiveness and effectiveness of pRMs through experiments in both finite and continuous physical environments, illustrating that incorporating pRMs significantly improves learning efficiency across several control tasks.

Related papers

• Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts [113.0656076371565]
  We propose a novel router-aware approach to optimize importance sampling weights in off-policy reinforcement learning (RL)<n>Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence.<n> Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models.
  arXiv  Detail & Related papers  (2025-10-27T05:47:48Z)
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS [62.22644307952087]
  We introduce AIRL-S, the first natural unification of RL-based and search-based TTS.<n>We leverage adversarial inverse reinforcement learning (AIRL) combined with group relative policy optimization (GRPO) to learn a dense, dynamic PRM directly from correct reasoning traces.<n>Our results show that our unified approach improves performance by 9 % on average over the base model, matching GPT-4o.
  arXiv  Detail & Related papers  (2025-08-19T23:41:15Z)
• Pushdown Reward Machines for Reinforcement Learning [17.63980224819404]
  We present pushdown reward machines (pdRMs), an extension of reward machines based on deterministic pushdown automata.<n>pdRMs can recognize and reward temporally extended behaviours representable in deterministic context-free languages.<n>We show how agents can be trained to perform tasks representable in deterministic context-free languages using pdRMs.
  arXiv  Detail & Related papers  (2025-08-09T08:59:09Z)
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner [31.033131727230277]
  Large reasoning models (LRMs) have recently shown promise in solving complex math problems when optimized with Reinforcement Learning (RL)<n>We propose a novel intrinsic signal-driven generative process evaluation mechanism operating at the thought level to address major bottlenecks in RL-based training.<n>Experiments on 1.5B and 7B parameter LRMs demonstrate that our method achieves higher problem-solving accuracy with significantly fewer training samples than outcome-only reward baselines.
  arXiv  Detail & Related papers  (2025-07-31T07:54:58Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe (Reinforced Learning) is a framework that combines the strengths of imitation learning and inverse reinforcement learning to learn a dense reward function efficiently.<n>Our framework produces high-performing policies in high-dimensional tasks where direct imitation fails to replicate complex behaviors.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation [0.7564784873669823]
  We propose Multimodal Contrastive Unsupervised Reinforcement Learning (M2CURL) Our approach employs a novel multimodal self-supervised learning technique that learns efficient representations and contributes to faster convergence of RL algorithms. We evaluate M2CURL on the Tactile Gym 2 simulator and we show that it significantly enhances the learning efficiency in different manipulation tasks.
  arXiv  Detail & Related papers  (2024-01-30T14:09:35Z)
• ASR: Attention-alike Structural Re-parameterization [53.019657810468026]
  We propose a simple-yet-effective attention-alike structural re- parameterization (ASR) that allows us to achieve SRP for a given network while enjoying the effectiveness of the attention mechanism. In this paper, we conduct extensive experiments from a statistical perspective and discover an interesting phenomenon Stripe Observation, which reveals that channel attention values quickly approach some constant vectors during training.
  arXiv  Detail & Related papers  (2023-04-13T08:52:34Z)
• Actively Learning Costly Reward Functions for Reinforcement Learning [56.34005280792013]
  We show that it is possible to train agents in complex real-world environments orders of magnitudes faster. By enabling the application of reinforcement learning methods to new domains, we show that we can find interesting and non-trivial solutions.
  arXiv  Detail & Related papers  (2022-11-23T19:17:20Z)
• Mask-based Latent Reconstruction for Reinforcement Learning [58.43247393611453]
  Mask-based Latent Reconstruction (MLR) is proposed to predict the complete state representations in the latent space from the observations with spatially and temporally masked pixels. Extensive experiments show that our MLR significantly improves the sample efficiency in deep reinforcement learning.
  arXiv  Detail & Related papers  (2022-01-28T13:07:11Z)
• MHER: Model-based Hindsight Experience Replay [33.00149668905828]
  We propose Model-based Hindsight Experience Replay (MHER) to solve multi-goal reinforcement learning problems. replacing original goals with virtual goals generated from interaction with a trained dynamics model leads to a novel relabeling method. MHER exploits experiences more efficiently by leveraging environmental dynamics to generate virtual achieved goals.
  arXiv  Detail & Related papers  (2021-07-01T08:52:45Z)
• Reinforcement Learning through Active Inference [62.997667081978825]
  We show how ideas from active inference can augment traditional reinforcement learning approaches. We develop and implement a novel objective for decision making, which we term the free energy of the expected future. We demonstrate that the resulting algorithm successfully exploration and exploitation, simultaneously achieving robust performance on several challenging RL benchmarks with sparse, well-shaped, and no rewards.
  arXiv  Detail & Related papers  (2020-02-28T10:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.