Conversational Code Generation: a Case Study of Designing a Dialogue System for Generating Driving Scenarios for Testing Autonomous Vehicles

• URL: http://arxiv.org/abs/2410.09829v2
• Date: Tue, 27 May 2025 09:29:18 GMT
• Title: Conversational Code Generation: a Case Study of Designing a Dialogue System for Generating Driving Scenarios for Testing Autonomous Vehicles
• Authors: Rimvydas Rubavicius, Antonio Valerio Miceli-Barone, Alex Lascarides, Subramanian Ramamoorthy,
• Abstract summary: We design a natural language interface to assist a non-coding domain expert in synthesising the desired scenarios and vehicle behaviours.<n>We show that using it to convert utterances to the symbolic program is feasible, despite the very small training dataset.<n>Human experiments show that dialogue is critical to successful simulation generation, leading to a 4.5 times higher success rate than a generation without engaging in extended conversation.
• Score: 20.757088470174452
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cyber-physical systems like autonomous vehicles are tested in simulation before deployment, using domain-specific programs for scenario specification. To aid the testing of autonomous vehicles in simulation, we design a natural language interface, using an instruction-following large language model, to assist a non-coding domain expert in synthesising the desired scenarios and vehicle behaviours. We show that using it to convert utterances to the symbolic program is feasible, despite the very small training dataset. Human experiments show that dialogue is critical to successful simulation generation, leading to a 4.5 times higher success rate than a generation without engaging in extended conversation.

Related papers

• LLM-based Realistic Safety-Critical Driving Video Generation [4.537331974356809]
  We propose a framework that automatically synthesizes driving scenarios within the CARLA simulator.<n>The framework has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics.<n>Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases.
  arXiv  Detail & Related papers  (2025-07-02T00:45:19Z)
• YuLan-OneSim: Towards the Next Generation of Social Simulator with Large Language Models [50.86336063222539]
  We introduce a novel social simulator called YuLan-OneSim.<n>Users can simply describe and refine their simulation scenarios through natural language interactions with our simulator.<n>We implement 50 default simulation scenarios spanning 8 domains, including economics, sociology, politics, psychology, organization, demographics, law, and communication.
  arXiv  Detail & Related papers  (2025-05-12T14:05:17Z)
• LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation [94.84458417662404]
  LangTraj is a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenarios. By conditioning on natural language inputs, LangTraj provides flexible and intuitive control over interactive behaviors. LangTraj demonstrates strong performance in realism, language controllability, and language-conditioned safety-critical simulation.
  arXiv  Detail & Related papers  (2025-04-15T17:14:06Z)
• BeSimulator: A Large Language Model Powered Text-based Behavior Simulator [28.112491177744783]
  We introduce BeSimulator as an attempt towards behavior simulation in the context of text-based environments. BeSimulator can generalize across scenarios and achieve long-horizon complex simulation.
  arXiv  Detail & Related papers  (2024-09-24T08:37:04Z)
• ChatSUMO: Large Language Model for Automating Traffic Scenario Generation in Simulation of Urban MObility [5.111204055180423]
  Large Language Models (LLMs) are capable of handling multi-modal input and outputs such as text, voice, images, and video. We present ChatSUMO, a LLM-based agent that integrates language processing skills to generate abstract and real-world simulation scenarios. For simulation generation, we created a real-world simulation for the city of Albany with an accuracy of 96%.
  arXiv  Detail & Related papers  (2024-08-29T03:59:11Z)
• On Languaging a Simulation Engine [6.17566001699186]
  Lang2Sim is a language-to-simulation framework that enables interactive navigation on languaging a simulation engine. This work establishes language model as an intelligent platform to unlock the era of languaging a simulation engine.
  arXiv  Detail & Related papers  (2024-02-26T11:01:54Z)
• RealGen: Retrieval Augmented Generation for Controllable Traffic Scenarios [58.62407014256686]
  RealGen is a novel retrieval-based in-context learning framework for traffic scenario generation. RealGen synthesizes new scenarios by combining behaviors from multiple retrieved examples in a gradient-free way. This in-context learning framework endows versatile generative capabilities, including the ability to edit scenarios.
  arXiv  Detail & Related papers  (2023-12-19T23:11:06Z)
• Dialogue-based generation of self-driving simulation scenarios using Large Language Models [14.86435467709869]
  Simulation is an invaluable tool for developing and evaluating controllers for self-driving cars. Current simulation frameworks are driven by highly-specialist domain specific languages. There is often a gap between a concise English utterance and the executable code that captures the user's intent.
  arXiv  Detail & Related papers  (2023-10-26T13:07:01Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• TrafficBots: Towards World Models for Autonomous Driving Simulation and Motion Prediction [149.5716746789134]
  We show data-driven traffic simulation can be formulated as a world model. We present TrafficBots, a multi-agent policy built upon motion prediction and end-to-end driving. Experiments on the open motion dataset show TrafficBots can simulate realistic multi-agent behaviors.
  arXiv  Detail & Related papers  (2023-03-07T18:28:41Z)
• Metaphorical User Simulators for Evaluating Task-oriented Dialogue Systems [80.77917437785773]
  Task-oriented dialogue systems ( TDSs) are assessed mainly in an offline setting or through human evaluation. We propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities.
  arXiv  Detail & Related papers  (2022-04-02T05:11:03Z)
• A Conversational Paradigm for Program Synthesis [110.94409515865867]
  We propose a conversational program synthesis approach via large language models. We train a family of large language models, called CodeGen, on natural language and programming language data. Our findings show the emergence of conversational capabilities and the effectiveness of the proposed conversational program synthesis paradigm.
  arXiv  Detail & Related papers  (2022-03-25T06:55:15Z)
• SilGAN: Generating driving maneuvers for scenario-based software-in-the-loop testing [0.0]
  SilGAN is a deep generative model that eases specification, stimulus generation, and automation of automotive software-in-the-loop testing. The model is trained using data recorded from vehicles in the field.
  arXiv  Detail & Related papers  (2021-07-05T07:17:49Z)
• SimNet: Learning Reactive Self-driving Simulations from Real-world Observations [10.035169936164504]
  We present an end-to-end trainable machine learning system capable of realistically simulating driving experiences. This can be used for the verification of self-driving system performance without relying on expensive and time-consuming road testing.
  arXiv  Detail & Related papers  (2021-05-26T05:14:23Z)
• DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
  We introduce a novel high-quality neural simulator referred to as DriveGAN. DriveGAN achieves controllability by disentangling different components without supervision. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
  arXiv  Detail & Related papers  (2021-04-30T15:30:05Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction [88.0416857308144]
  We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps. We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
  arXiv  Detail & Related papers  (2020-08-13T17:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.