PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation

• URL: http://arxiv.org/abs/2606.12616v1
• Date: Wed, 10 Jun 2026 19:16:31 GMT
• Title: PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation
• Abstract summary: We introduce PersonaDrive, a pipeline that conditions a vision--action driving agent on retrieved demonstrations from a style-instructed human driving dataset.<n>The pipeline has three stages: offline triplet mining over per-style human driving data using a combined image-text similarity score; training a lightweight retrieval head that fuses frozen visual features with a small control encoder over per-style databases; and fine-tuning a single VLA backbone to treat retrieved context points as behavioral demonstrations.
• Score: 9.651475921652052
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Closed-loop driving simulators typically populate their environments with non-ego traffic agents that behave largely the same way, produced either by rule-based traffic managers or by learned models trained toward a single behavioral mode. Recent work introduces style variation through post-hoc labels on observational data or LLM-inferred reward weights, but these signals act as proxies for what a style should reward rather than demonstrations of humans explicitly asked to drive in that style. We introduce PersonaDrive, a pipeline that conditions a vision-language-action (VLA) driving agent on retrieved demonstrations from a style-instructed human driving dataset, in which participants drive CARLA leaderboard routes under aggressive, neutral, and conservative instructions on a driver-in-the-loop rig. The pipeline has three stages: (i) offline triplet mining over per-style human driving data using a combined image-text similarity score; (ii) training a lightweight retrieval head that fuses frozen visual features with a small control encoder over per-style databases; and (iii) fine-tuning a single VLA backbone to treat retrieved context points as in-context behavioral demonstrations during waypoint prediction. At inference, the same backbone is conditioned on any style by swapping which per-style database the retrieval head queries, so selecting a style requires no per-style retraining while enabling human-style, style-diverse non-ego agents for closed-loop simulation. On Bench2Drive, PersonaDrive (no style) improves the driving score by 4.6% over SimLingo and 2.5% over HiP-AD, and under style conditioning attains the highest driving score in every style within a roughly 2% band (its weakest style surpassing the strongest baseline, DMW, by 5.4%), while average speed and acceleration rise by 18% and 25% from the conservative to the aggressive instruction.

Related papers

• SkyDrive: Learning to Drive in a New City from Aerial Traffic Monitoring [69.91458772455071]
  emphSkyDrive is a framework that utilizes drone-based traffic monitoring to provide efficient supervision for autonomous driving agents in a new environment.<n>Our findings show that aerial traffic monitoring is an efficient and scalable data source for adapting autonomous driving systems in new cities.
  arXiv  Detail & Related papers  (2026-08-25T20:47:09Z)
• PersonaDrive: Controllable Trajectory Prediction with Multi-Dimensional Driving Personas [24.951167128527477]
  We present the Persona-Conditioned Trajectory dataset, which decomposes driving personas along two axes, Temporal Urgency and Ride Comfort.<n>We also present PersonaDrive, a framework that can learn driving personas from language and can generate persona-specific trajectories.<n>Tests show that PersonaDrive consistently improves over the compared baselines across multi-dimensional scenarios.
  arXiv  Detail & Related papers  (2026-08-15T13:37:40Z)
• STAGE: STyle-controllable Action GEneration for personalized autonomous driving [15.187069954986343]
  Driving style refers to the behavioral preferences that drivers maintain during driving.<n>We develop a style controllable action generation method, STAGE, for driving tasks.<n>We reveal the unique functionalities of STAGE, including its style controllability, style continuity, driving style alignment capability and driving safety.
  arXiv  Detail & Related papers  (2026-07-31T15:17:50Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• GSDrive: Reinforcing Driving Policies by Multi-mode Trajectory Probing with 3D Gaussian Splatting Environment [6.425301529118419]
  End-to-end (E2E) autonomous driving presents a promising approach for translating perceptual inputs directly into driving actions.<n>We introduce GSDrive, a framework that exploits 3D Gaussian Splatting (3DGS) for differentiable, physics-based reward shaping in E2E driving policy improvement.
  arXiv  Detail & Related papers  (2026-04-30T16:59:07Z)
• Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models [50.22099309218635]
  In autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation.<n>We present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving.
  arXiv  Detail & Related papers  (2026-04-01T11:38:46Z)
• StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving [8.903468887918754]
  StyleVLA is a physics-informed VLA framework for generating diverse and physically plausible driving behaviors.<n>We construct a large-scale instruction dataset with over 1.2k scenarios, 76k Bird's Eye View (BEV) samples, and 42k First Person View (FPV) samples.<n>Experiments show that StyleVLA significantly outperforms proprietary models and state-of-the-art VLA models.
  arXiv  Detail & Related papers  (2026-03-10T10:33:58Z)
• RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies [30.632104005565832]
  Rollouts as Demonstrations (RoaD) is a method to mitigate covariate shift when training autonomous driving policies in closed loop.<n>During rollout generation, RoaD incorporates expert guidance to bias trajectories toward high-quality behavior, producing informative yet realistic demonstrations for fine-tuning.<n>We demonstrate the effectiveness of RoaD on WOSAC, a large-scale traffic simulation benchmark, where it performs similar or better than the prior CL-SFT method.
  arXiv  Detail & Related papers  (2025-12-01T18:52:03Z)
• TrafficBots V1.5: Traffic Simulation via Conditional VAEs and Transformers with Relative Pose Encoding [59.339735703856924]
  TrafficBots V1.5 is a baseline method for the closed-loop simulation of traffic agents. It achieves a 3rd place ranking in the Open Sim Agents Challenge (WOSAC) 2024.
  arXiv  Detail & Related papers  (2024-06-16T11:20:49Z)
• Situation Awareness for Driver-Centric Driving Style Adaptation [3.568617847600189]
  We propose a situation-aware driving style model based on different visual feature encoders pretrained on fleet data. Our experiments show that the proposed method outperforms static driving styles significantly and forms plausible situation clusters.
  arXiv  Detail & Related papers  (2024-03-28T17:19:16Z)
• Driver Profiling and Bayesian Workload Estimation Using Naturalistic Peripheral Detection Study Data [40.43737902900321]
  We tackle the problem of workload estimation from driving performance data. Key environmental factors that induce a high mental workload are identified via video analysis. A supervised learning framework is introduced to profile drivers based on the average workload they experience. A Bayesian filtering approach is then proposed for sequentially estimating, in (near) real-time, the driver's instantaneous workload.
  arXiv  Detail & Related papers  (2023-03-26T13:15:44Z)
• Policy Pre-training for End-to-end Autonomous Driving via Self-supervised Geometric Modeling [96.31941517446859]
  We propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
  arXiv  Detail & Related papers  (2023-01-03T08:52:49Z)
• Learning to drive from a world on rails [78.28647825246472]
  We learn an interactive vision-based driving policy from pre-recorded driving logs via a model-based approach. A forward model of the world supervises a driving policy that predicts the outcome of any potential driving trajectory. Our method ranks first on the CARLA leaderboard, attaining a 25% higher driving score while using 40 times less data.
  arXiv  Detail & Related papers  (2021-05-03T05:55:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.