Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies

• URL: http://arxiv.org/abs/2606.08312v1
• Date: Sat, 06 Jun 2026 19:55:54 GMT
• Title: Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policies
• Abstract summary: We study offline reinforcement learning (RL) under temporally extended task constraints expressed in Linear Temporal Logic over finite traces (LTLf)<n>We introduce a neurosymbolic framework that injectsf background knowledge into such transformer-based RL policies.<n> Experimental results show that incorporating background knowledge not only improves constraint satisfaction, but also maintains competitive return.
• Score: 5.081811811279625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work we study offline reinforcement learning (RL) under temporally extended task constraints expressed in Linear Temporal Logic over finite traces (LTLf). Recently, transformer-based approaches such as Trajectory Transformers and Decision Transformers have been adopted to address RL as a sequence modeling problem. However, these methods optimize purely for reward and do not account for high-level temporal requirements. Here, we introduce a neurosymbolic framework that injects LTLf background knowledge into such transformer-based RL policies. Our approach compiles LTLf formulas into deterministic finite automata (DFAs) and integrates them into the learning process through a differentiable representation and a logic-based loss function. In particular, we derive differentiable satisfaction signals from DFA progression and use them as a regularization term during training. The resulting method is architecture-agnostic across different models. We evaluate the proposed framework on navigation environments with specification suites covering combinations of safety and reachability temporal properties. Experimental results show that incorporating background knowledge not only improves constraint satisfaction, but also maintains competitive return compared to vanilla baselines.

Related papers

• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement [23.24714915251971]
  We investigate the ability of transformers to perform in-context reinforcement learning.<n>We show that a linear self-attention transformer block can provably implement policy-improvement methods.<n>We establish the first convergence guarantee in the ICRL literature.
  arXiv  Detail & Related papers  (2026-05-07T06:50:49Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Control Synthesis of Cyber-Physical Systems for Real-Time Specifications through Causation-Guided Reinforcement Learning [3.608670495432032]
  Signal temporal logic (STL) has emerged as a powerful formalism of expressing real-time constraints.<n> reinforcement learning (RL) has become an important method for solving control synthesis problems in unknown environments.<n>We propose an online reward generation method guided by the online causation monitoring of STL.
  arXiv  Detail & Related papers  (2025-10-09T02:49:28Z)
• Formally Verified Neurosymbolic Trajectory Learning via Tensor-based Linear Temporal Logic on Finite Traces [0.0]
  We present a novel formalisation of tensor semantics for linear temporal logic on finite traces (LTLf)<n>We demonstrate that this formalisation can be integrated into a neurosymbolic learning process by defining and verifying a differentiable loss function for the constraints.<n>We show that, by using this loss, the process learns to satisfy pre-specified logical constraints.
  arXiv  Detail & Related papers  (2025-01-23T14:43:12Z)
• Directed Exploration in Reinforcement Learning from Linear Temporal Logic [59.707408697394534]
  Linear temporal logic (LTL) is a powerful language for task specification in reinforcement learning.<n>We show that the synthesized reward signal remains fundamentally sparse, making exploration challenging.<n>We show how better exploration can be achieved by further leveraging the specification and casting its corresponding Limit Deterministic B"uchi Automaton (LDBA) as a Markov reward process.
  arXiv  Detail & Related papers  (2024-08-18T14:25:44Z)
• Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning [7.811937592466619]
  We propose temporal logic Specification-conditioned Decision Transformer (SDT)<n>SDT harnesses the expressive power of signal temporal logic to specify complex temporal rules that an agent should follow.<n> Empirical evaluations on the DSRL benchmarks demonstrate the better capacity of SDT in learning safe and high-reward policies.
  arXiv  Detail & Related papers  (2024-02-27T05:16:59Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Rethinking Decision Transformer via Hierarchical Reinforcement Learning [54.3596066989024]
  Decision Transformer (DT) is an innovative algorithm leveraging recent advances of the transformer architecture in reinforcement learning (RL) We introduce a general sequence modeling framework for studying sequential decision making through the lens of Hierarchical RL. We show DT emerges as a special case of this framework with certain choices of high-level and low-level policies, and discuss the potential failure of these choices.
  arXiv  Detail & Related papers  (2023-11-01T03:32:13Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations [0.34410212782758043]
  Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations.
  arXiv  Detail & Related papers  (2023-07-06T12:33:34Z)
• Certified Reinforcement Learning with Logic Guidance [78.2286146954051]
  We propose a model-free RL algorithm that enables the use of Linear Temporal Logic (LTL) to formulate a goal for unknown continuous-state/action Markov Decision Processes (MDPs) The algorithm is guaranteed to synthesise a control policy whose traces satisfy the specification with maximal probability.
  arXiv  Detail & Related papers  (2019-02-02T20:09:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.