VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving

• URL: http://arxiv.org/abs/2412.15544v1
• Date: Fri, 20 Dec 2024 04:08:11 GMT
• Title: VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Driving
• Authors: Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, Sikai Chen,
• Abstract summary: Reinforcement learning (RL)-based methods for learning driving policies have gained increasing attention in the autonomous driving community. Traditional RL approaches rely on manually engineered rewards, which require extensive human effort and often lack generalizability. We propose textbfVLM-RL, a unified framework that integrates pre-trained Vision-Language Models (VLMs) with RL to generate reward signals.
• Score: 1.3107174618549584
• License:
• Abstract: In recent years, reinforcement learning (RL)-based methods for learning driving policies have gained increasing attention in the autonomous driving community and have achieved remarkable progress in various driving scenarios. However, traditional RL approaches rely on manually engineered rewards, which require extensive human effort and often lack generalizability. To address these limitations, we propose \textbf{VLM-RL}, a unified framework that integrates pre-trained Vision-Language Models (VLMs) with RL to generate reward signals using image observation and natural language goals. The core of VLM-RL is the contrasting language goal (CLG)-as-reward paradigm, which uses positive and negative language goals to generate semantic rewards. We further introduce a hierarchical reward synthesis approach that combines CLG-based semantic rewards with vehicle state information, improving reward stability and offering a more comprehensive reward signal. Additionally, a batch-processing technique is employed to optimize computational efficiency during training. Extensive experiments in the CARLA simulator demonstrate that VLM-RL outperforms state-of-the-art baselines, achieving a 10.5\% reduction in collision rate, a 104.6\% increase in route completion rate, and robust generalization to unseen driving scenarios. Furthermore, VLM-RL can seamlessly integrate almost any standard RL algorithms, potentially revolutionizing the existing RL paradigm that relies on manual reward engineering and enabling continuous performance improvements. The demo video and code can be accessed at: https://zilin-huang.github.io/VLM-RL-website.

Related papers

• Large Language Model-Enhanced Reinforcement Learning for Generic Bus Holding Control Strategies [12.599164162404994]
  This study introduces an automatic reward generation paradigm by leveraging the in-context learning and reasoning capabilities of Large Language Models (LLMs) To evaluate the feasibility of the proposed LLM-enhanced RL paradigm, it is applied to various bus holding control scenarios, including a synthetic single-line system and a real-world multi-line system.
  arXiv  Detail & Related papers  (2024-10-14T07:10:16Z)
• Generating and Evolving Reward Functions for Highway Driving with Large Language Models [18.464822261908562]
  Reinforcement Learning (RL) plays a crucial role in advancing autonomous driving technologies. We introduce a novel framework that integrates Large Language Models (LLMs) with RL to improve reward function design in autonomous driving.
  arXiv  Detail & Related papers  (2024-06-15T07:50:10Z)
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL [80.10358123795946]
  We develop a framework for building multi-turn RL algorithms for fine-tuning large language models. Our framework adopts a hierarchical RL approach and runs two RL algorithms in parallel. Empirically, we find that ArCHer significantly improves efficiency and performance on agent tasks.
  arXiv  Detail & Related papers  (2024-02-29T18:45:56Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• Code as Reward: Empowering Reinforcement Learning with VLMs [37.862999288331906]
  We propose a framework named Code as Reward (VLM-CaR) to produce dense reward functions from pre-trained Vision-Language Models. VLM-CaR significantly reduces the computational burden of querying the VLM directly. We show that the dense rewards generated through our approach are very accurate across a diverse set of discrete and continuous environments.
  arXiv  Detail & Related papers  (2024-02-07T11:27:45Z)
• Language Reward Modulation for Pretraining Reinforcement Learning [61.76572261146311]
  We propose leveraging the capabilities of LRFs as a pretraining signal for reinforcement learning. Our VLM pretraining approach, which is a departure from previous attempts to use LRFs, can warmstart sample-efficient learning on robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-08-23T17:37:51Z)
• Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization [73.74371798168642]
  We introduce an open-source modular library, RL4LMs, for optimizing language generators with reinforcement learning. Next, we present the GRUE benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO, that learns to effectively reduce the action space in language generation.
  arXiv  Detail & Related papers  (2022-10-03T21:38:29Z)
• Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning [13.699336307578488]
  Deep imitative reinforcement learning approach (DIRL) achieves agile autonomous racing using visual inputs. We validate our algorithm both in a high-fidelity driving simulation and on a real-world 1/20-scale RC-car with limited onboard computation.
  arXiv  Detail & Related papers  (2021-07-18T00:00:48Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real [74.45688231140689]
  We introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. We obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T08:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.