CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs

• URL: http://arxiv.org/abs/2607.04854v1
• Date: Mon, 06 Jul 2026 09:29:57 GMT
• Title: CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs
• Abstract summary: Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications.<n>This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process.<n>We propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs' intrinsic focus on constraints.
• Score: 25.005954937023052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process. While existing approaches attempt to mitigate this by relying on external tools or task decomposition, they fail to enhance the model's intrinsic constraint awareness. To address this, we propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs' intrinsic focus on constraints. CARL introduces a constraint-aware reward by comparing the model's output distributions under constrained and unconstrained inputs, encouraging constraint focus and penalizing neglect. Compatible with various RL frameworks and requiring no external solvers or top models, CARL enables scalable, end-to-end constraint-aware planning. Extensive experiments on BlocksWorld, TravelPlanner, and T-Eval demonstrate that CARL significantly outperforms standard Reinforcement Fine-Tuning (RFT) baselines and state-of-the-art reasoning models, exhibiting a markedly increased focus on constraints.

Related papers

• Bridging Auxiliary Constraints to Resolve Instruction Following in Large Reasoning Models [52.87406450655783]
  Large Reasoning Models (LRMs) have demonstrated impressive capabilities in many tasks, yet they struggle with reliably following multiple instructions.<n>We formalize this challenge as the Constraint Adherence Problem (CAP)<n>This paper introduces a novel framework that addresses CAP by representing instructions as a structured knowledge graph of constraints.
  arXiv  Detail & Related papers  (2026-06-02T13:23:28Z)
• AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution [61.593935260052795]
  Post-training viaSupervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs)<n>Existing paradigms often reach a performance bottleneck due to the limitations of static data.<n>We propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution.
  arXiv  Detail & Related papers  (2026-05-25T08:26:34Z)
• Reinforcing Structured Chain-of-Thought for Video Understanding [49.3086326382764]
  We introduce Summary-Driven Reinforcement Learning (SDRL)<n>SDRL is a single-stage RL framework that obviates the need forSupervised Fine-Tuning (SFT)<n>Our method achieves state-of-the-art performance on seven public VideoQA datasets.
  arXiv  Detail & Related papers  (2026-03-26T22:11:14Z)
• Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner [15.619469979987429]
  Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs.<n>We propose textitdynamic constraints that adapt to the evolving capabilities of the fine-tuned model.<n>Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines.
  arXiv  Detail & Related papers  (2026-03-18T08:37:31Z)
• PILOT: Planning via Internalized Latent Optimization Trajectories for Large Language Models [51.43746425777865]
  Large Language Models (LLMs) often lack the capacity to formulate global strategies, leading to error propagation in long-horizon tasks.<n>We propose PILOT, a framework designed to internalize the strategic oversight of large models into intrinsic Latent Guidance.
  arXiv  Detail & Related papers  (2026-01-07T12:38:56Z)
• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning [52.03884701766989]
  offline reinforcement learning (RL) algorithms typically impose constraints on action selection.<n>We propose a new neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods of dataset actions.<n>We develop a simple yet effective algorithm, Adaptive Neighborhood-constrained Q learning (ANQ), to perform Q learning with target actions satisfying this constraint.
  arXiv  Detail & Related papers  (2025-11-04T13:42:05Z)
• ConstraintLLM: A Neuro-Symbolic Framework for Industrial-Level Constraint Programming [9.69748612176497]
  Constraint programming (CP) is a crucial technology for solving real-world constraint optimization problems (COPs)<n>Using large language models (LLMs) to generate formal modeling automatically for COPs is becoming a promising approach, which aims to build neuro-symbolic AI with the help of symbolic solvers.<n>We introduce ConstraintLLM, the first LLM specifically designed for CP modeling, which is trained on an open-source LLM with multi-instruction supervised fine-tuning.<n>We construct and release IndusCP, the first industrial-level benchmark for CP modeling, which contains 140 challenging tasks from various domains
  arXiv  Detail & Related papers  (2025-10-07T10:43:39Z)
• POT: Inducing Overthinking in LLMs via Black-Box Iterative Optimization [28.771942726400084]
  We propose POT (Prompt-Only OverThinking), a black-box attack framework that employs iterative optimization to generate semantically natural adversarial prompts.<n>PoT achieves superior performance compared to other methods.
  arXiv  Detail & Related papers  (2025-08-23T16:27:42Z)
• R-ConstraintBench: Evaluating LLMs on NP-Complete Scheduling [0.0]
  We present R-ConstraintBench, a framework that evaluates models on Resource-Constrained Project Scheduling Problems (RCPSP)<n>We instantiate the benchmark in a data center migration setting and evaluate multiple LLMs using feasibility and error analysis.<n> Empirically, strong models are near-ceiling on precedence-only DAGs, but feasibility performance collapses when downtime, temporal windows, and disjunctive constraints interact.
  arXiv  Detail & Related papers  (2025-08-21T03:35:58Z)
• Scalable Chain of Thoughts via Elastic Reasoning [61.75753924952059]
  Elastic Reasoning is a novel framework for scalable chain of thoughts.<n>It separates reasoning into two phases--thinking and solution--with independently allocated budgets.<n>Our approach produces more concise and efficient reasoning even in unconstrained settings.
  arXiv  Detail & Related papers  (2025-05-08T15:01:06Z)
• Navigating Demand Uncertainty in Container Shipping: Deep Reinforcement Learning for Enabling Adaptive and Feasible Master Stowage Planning [3.565151496245487]
  In this work, we address sequential dynamic decision-making problems with state-dependent constraints.<n>As a relevant and real-world case study, we focus on the stowage planning problem in container shipping.<n>We propose a deep RL framework with an encoder-decoder model and feasibility layers that satisfy convex constraints and maintain unbiased gradient flow.
  arXiv  Detail & Related papers  (2025-02-18T11:18:17Z)
• Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification [76.14641982122696]
  We propose a constraint learning schema for fine-tuning Large Language Models (LLMs) with attribute control. We show that our approach leads to an LLM that produces fewer inappropriate responses while achieving competitive performance on benchmarks and a toxicity detection task.
  arXiv  Detail & Related papers  (2024-10-07T23:38:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.