An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordination

• URL: http://arxiv.org/abs/2504.19480v1
• Date: Mon, 28 Apr 2025 04:41:15 GMT
• Title: An Automated Reinforcement Learning Reward Design Framework with Large Language Model for Cooperative Platoon Coordination
• Abstract summary: Reinforcement Learning (RL) has demonstrated excellent decision-making potential in platoon coordination problems.<n>Finding a well performance reward function to guide RL training to solve complex platoon coordination problems remains challenging.<n>We propose a Large Language Model (LLM)-based Platoon coordination Reward Design (PCRD) framework, which systematically automates reward function discovery.
• Score: 10.669043457982404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) has demonstrated excellent decision-making potential in platoon coordination problems. However, due to the variability of coordination goals, the complexity of the decision problem, and the time-consumption of trial-and-error in manual design, finding a well performance reward function to guide RL training to solve complex platoon coordination problems remains challenging. In this paper, we formally define the Platoon Coordination Reward Design Problem (PCRDP), extending the RL-based cooperative platoon coordination problem to incorporate automated reward function generation. To address PCRDP, we propose a Large Language Model (LLM)-based Platoon coordination Reward Design (PCRD) framework, which systematically automates reward function discovery through LLM-driven initialization and iterative optimization. In this method, LLM first initializes reward functions based on environment code and task requirements with an Analysis and Initial Reward (AIR) module, and then iteratively optimizes them based on training feedback with an evolutionary module. The AIR module guides LLM to deepen their understanding of code and tasks through a chain of thought, effectively mitigating hallucination risks in code generation. The evolutionary module fine-tunes and reconstructs the reward function, achieving a balance between exploration diversity and convergence stability for training. To validate our approach, we establish six challenging coordination scenarios with varying complexity levels within the Yangtze River Delta transportation network simulation. Comparative experimental results demonstrate that RL agents utilizing PCRD-generated reward functions consistently outperform human-engineered reward functions, achieving an average of 10\% higher performance metrics in all scenarios.

Related papers

• Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation [23.371552518874807]
  Automated Reward Design (ARD) aims to replace manual reward engineering in reinforcement learning with language-driven reward function synthesis.<n>Existing approaches based on large language models (LLMs) remain inherently correlation-driven, relying on iterative environmental feedback to refine reward hypotheses for each specific task.<n>We propose the Causal Reward World Model (CRWM), which explicitly models the causal relationships between candidate reward components and task-targeted physical variables through offline pre-training on multi-task interaction data.
  arXiv  Detail & Related papers  (2026-06-22T12:57:17Z)
• Smart Commander: A Hierarchical Reinforcement Learning Framework for Fleet-Level PHM Decision Optimization [6.029264044194714]
  This paper proposes Smart Commander, a novel Hierarchical Reinforcement Learning (HRL) framework to optimize sequential maintenance and logistics decisions.<n>The framework decomposes the complex control problem into a two-tier hierarchy: a strategic General Commander manages fleet-level availability and cost objectives, while tactical Operation Commanders execute specific actions for sortie generation, maintenance scheduling, and resource allocation.<n> Empirical evaluations demonstrate that Smart Commander significantly outperforms conventional monolithic DeepReinforcement Learning (DRL) and rule-based baselines.
  arXiv  Detail & Related papers  (2026-04-08T15:00:12Z)
• Differentiable Evolutionary Reinforcement Learning [41.96953381133274]
  We propose Differentiable Evolutionary Reinforcement Learning (DERL), a bilevel framework that enables the autonomous discovery of optimal reward signals.<n>DERL is differentiable in its metaoptimization: it treats the inner-loop validation performance as a signal to update the Meta-r via reinforcement learning.<n> Experimental results show that DERL achieves state-of-the-art performance on ALFWorld and ScienceWorld, significantly outperforming methods relying on rewards.
  arXiv  Detail & Related papers  (2025-12-15T14:50:08Z)
• Multi-agent In-context Coordination via Decentralized Memory Retrieval [39.106914463842685]
  Large transformer models, trained on diverse datasets, have demonstrated impressive few-shot performance on previously unseen tasks.<n>In cooperative Multi-Agent Reinforcement Learning (MARL), where agents must coordinate toward a shared goal, decentralized policy deployment can lead to mismatches in task alignment and reward assignment.<n>We introduce Multi-agent In-context Coordination via Decentralized Memory Retrieval (MAICC), a novel approach designed to enhance coordination by fast adaptation.
  arXiv  Detail & Related papers  (2025-11-13T07:08:31Z)
• An End-to-End Deep Reinforcement Learning Approach for Solving the Traveling Salesman Problem with Drones [12.385878815004283]
  This study proposes a hierarchical Actor-Critic deep reinforcement learning framework for solving the Traveling Salesman Problem with Drones (TSP-D)<n>The architecture consists of two primary computation: a Transformer-inspired encoder and an efficient Minimal Gated Unit decoder.<n>The entire framework operates within an asynchronous advantage actor-critic paradigm.
  arXiv  Detail & Related papers  (2025-11-07T14:26:29Z)
• Closing the Loop: Coordinating Inventory and Recommendation via Deep Reinforcement Learning on Multiple Timescales [6.5434300333872875]
  This paper proposes a unified multi-agent RL framework tailored for joint optimization across distinct functional modules.<n>We first develop an integrated theoretical model to capture the interplay between these functions.<n>We then design a novel multi-timescale multi-agent RL architecture that decomposes policy components according to departmental functions.
  arXiv  Detail & Related papers  (2025-10-05T16:28:06Z)
• Causal-Paced Deep Reinforcement Learning [4.728991543521559]
  Causal-Paced Deep Reinforcement Learning (CP-DRL) is a curriculum learning framework aware of SCM differences between tasks based on interaction data approximation.<n> Empirically, CP-DRL outperforms existing curriculum methods on the Point Mass benchmark.
  arXiv  Detail & Related papers  (2025-06-24T20:15:01Z)
• RL for Reasoning by Adaptively Revealing Rationales [36.50924054394857]
  Supervised fine-tuning (SFT) relies on dense ground-truth labels, which become increasingly costly as sequence length grows.<n>We address this by adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals only a partial prefix of the target output during training.<n>We show that our adaptive curriculum over partial answers reliably solves problems that are otherwise intractable.
  arXiv  Detail & Related papers  (2025-06-22T17:46:14Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning Tasks [110.20297293596005]
  Large language model (LLM) agents need to perform multi-turn interactions in real-world tasks. Existing multi-turn RL algorithms for optimizing LLM agents fail to perform effective credit assignment over multiple turns while leveraging the generalization capabilities of LLMs. We propose a novel RL algorithm, SWEET-RL, that uses a carefully designed optimization objective to train a critic model with access to additional training-time information. Our experiments demonstrate that SWEET-RL achieves a 6% absolute improvement in success and win rates on ColBench compared to other state-of-the-art multi-turn RL algorithms.
  arXiv  Detail & Related papers  (2025-03-19T17:55:08Z)
• Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs [58.18140409409302]
  Large Language Models (LLMs) have made substantial strides in structured tasks through Reinforcement Learning (RL) Applying RL in broader domains like chatbots and content generation presents unique challenges. We show a case study of reproducing existing reward model ensemble research using embedding-based reward models.
  arXiv  Detail & Related papers  (2025-02-04T19:37:35Z)
• Adaptive Reward Design for Reinforcement Learning in Complex Robotic Tasks [2.3031174164121127]
  We propose a suite of reward functions that incentivize an RL agent to make measurable progress on tasks specified by formulas.<n>We develop an adaptive reward shaping approach that dynamically updates these reward functions during the learning process.<n> Experimental results on a range of RL-based robotic tasks demonstrate that the proposed approach is compatible with various RL algorithms.
  arXiv  Detail & Related papers  (2024-12-14T18:04:18Z)
• Enhancing Spectrum Efficiency in 6G Satellite Networks: A GAIL-Powered Policy Learning via Asynchronous Federated Inverse Reinforcement Learning [67.95280175998792]
  A novel adversarial imitation learning (GAIL)-powered policy learning approach is proposed for optimizing beamforming, spectrum allocation, and remote user equipment (RUE) association ins. We employ inverse RL (IRL) to automatically learn reward functions without manual tuning. We show that the proposed MA-AL method outperforms traditional RL approaches, achieving a $14.6%$ improvement in convergence and reward value.
  arXiv  Detail & Related papers  (2024-09-27T13:05:02Z)
• Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs [12.572869123617783]
  Reinforcement learning (RL) faces challenges in evaluating policy trajectories within intricate game tasks. PbRL presents a pioneering framework that capitalizes on human preferences as pivotal reward signals. We propose a LLM-enabled automatic preference generation framework named LLM4PG.
  arXiv  Detail & Related papers  (2024-06-28T04:21:24Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks [63.24965775030673]
  We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
  arXiv  Detail & Related papers  (2023-02-02T18:27:20Z)
• Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Training [43.97565851415018]
  Robust Reinforcement Learning (RL) focuses on improving performances under model errors or adversarial attacks. Most of the existing literature models RARL as a zero-sum simultaneous game with Nash equilibrium as the solution concept. We introduce a novel hierarchical formulation of robust RL - a general-sum Stackelberg game model called RRL-Stack.
  arXiv  Detail & Related papers  (2022-02-19T03:44:05Z)
• Collision-Free Flocking with a Dynamic Squad of Fixed-Wing UAVs Using Deep Reinforcement Learning [2.555094847583209]
  We deal with the decentralized leader-follower flocking control problem through deep reinforcement learning (DRL) We propose a novel reinforcement learning algorithm CACER-II for training a shared control policy for all the followers. As a result, the variable-length system state can be encoded into a fixed-length embedding vector, which makes the learned DRL policies independent with the number or the order of followers.
  arXiv  Detail & Related papers  (2021-01-20T11:23:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.