A Safe Preference Learning Approach for Personalization with Applications to Autonomous Vehicles

• URL: http://arxiv.org/abs/2311.02099v4
• Date: Tue, 26 Mar 2024 14:25:52 GMT
• Title: A Safe Preference Learning Approach for Personalization with Applications to Autonomous Vehicles
• Abstract summary: This work introduces a preference learning method that ensures adherence to given specifications, with an application to autonomous vehicles. We formulate the problem of safety-guaranteed preference learning based on pairwise comparisons and propose an approach to solve this learning problem. Our approach yields competitive results compared to existing preference learning methods in terms of capturing preferences and notably outperforms them when safety is considered.
• Score: 1.5553847214012173
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work introduces a preference learning method that ensures adherence to given specifications, with an application to autonomous vehicles. Our approach incorporates the priority ordering of Signal Temporal Logic (STL) formulas describing traffic rules into a learning framework. By leveraging Parametric Weighted Signal Temporal Logic (PWSTL), we formulate the problem of safety-guaranteed preference learning based on pairwise comparisons and propose an approach to solve this learning problem. Our approach finds a feasible valuation for the weights of the given PWSTL formula such that, with these weights, preferred signals have weighted quantitative satisfaction measures greater than their non-preferred counterparts. The feasible valuation of weights given by our approach leads to a weighted STL formula that can be used in correct-and-custom-by-construction controller synthesis. We demonstrate the performance of our method with a pilot human subject study in two different simulated driving scenarios involving a stop sign and a pedestrian crossing. Our approach yields competitive results compared to existing preference learning methods in terms of capturing preferences and notably outperforms them when safety is considered.

Related papers

• Global stability of vehicle-with-driver dynamics via Sum-of-Squares programming [0.0]
  This work estimates safe in subsets of the Region of Attraction (ROA) for a seven-state vehicle-with-driver system.<n>Safe sets are computed by optimizing Lyapunov functions through an original iterative Sum-of-Squares (SOS) procedure.
  arXiv  Detail & Related papers  (2025-12-05T15:27:52Z)
• Feature-Guided SAE Steering for Refusal-Rate Control using Contrasting Prompts [1.1565815257603067]
  Large Language Model (LLM) deployment requires guiding the LLM to recognize and not answer unsafe prompts while complying with safe prompts.<n>Previous methods for achieving this require adjusting model weights along with other expensive procedures.<n>We explore using different steering features and steering strengths using Sparse Autoencoders (SAEs) to provide a solution.
  arXiv  Detail & Related papers  (2025-10-26T20:54:30Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• LLM Routing with Dueling Feedback [49.67815163970033]
  We study the problem of selecting the best model for each query while balancing user satisfaction, model expertise, and inference cost.<n>We formulate routing as contextual dueling bandits, learning from pairwise preference feedback rather than absolute scores.<n>We introduce Category-Calibrated Fine-Tuning (CCFT), a representation-learning method that derives model embeddings from offline data using contrastive fine-tuning with categorical weighting.
  arXiv  Detail & Related papers  (2025-10-01T12:52:25Z)
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving [55.13109926181247]
  We introduce ReflectDrive, a learning-based framework that integrates a reflection mechanism for safe trajectory generation via discrete diffusion.<n>Central to our approach is a safety-aware reflection mechanism that performs iterative self-correction without gradient.<n>Our method begins with goal-conditioned trajectory generation to model multi-modal driving behaviors.
  arXiv  Detail & Related papers  (2025-09-24T13:35:15Z)
• A Framework for Learning Scoring Rules in Autonomous Driving Planning Systems [2.4578723416255754]
  FLoRA is a framework that learns interpretable scoring rules represented in temporal logic. Our approach effectively learns to evaluate driving behavior even though the training data only contains positive examples. Evaluations in closed-loop planning simulations demonstrate that our learned scoring rules outperform existing techniques.
  arXiv  Detail & Related papers  (2025-02-17T02:06:57Z)
• Collision Probability Distribution Estimation via Temporal Difference Learning [0.46085106405479537]
  We introduce CollisionPro, a pioneering framework designed to estimate cumulative collision probability distributions. We formulate our framework within the context of reinforcement learning to pave the way for safety-aware agents. A comprehensive examination of our framework is conducted using a realistic autonomous driving simulator.
  arXiv  Detail & Related papers  (2024-07-29T13:32:42Z)
• Nash Learning from Human Feedback [86.09617990412941]
  We introduce an alternative pipeline for the fine-tuning of large language models using pairwise human feedback. We term this approach Nash learning from human feedback (NLHF) We present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent.
  arXiv  Detail & Related papers  (2023-12-01T19:26:23Z)
• Integrating Higher-Order Dynamics and Roadway-Compliance into Constrained ILQR-based Trajectory Planning for Autonomous Vehicles [3.200238632208686]
  Trajectory planning aims to produce a globally optimal route for Autonomous Passenger Vehicles. Existing implementations utilizing the vehicle bicycle kinematic model may not guarantee controllable trajectories. We augment this model by higher-order terms, including the first and second-order derivatives of curvature and longitudinal jerk.
  arXiv  Detail & Related papers  (2023-09-25T22:30:18Z)
• Actively Learning Reinforcement Learning: A Stochastic Optimal Control Approach [3.453622106101339]
  We propose a framework towards achieving two intertwined objectives: (i) equipping reinforcement learning with active exploration and deliberate information gathering, and (ii) overcoming the computational intractability of optimal control law. We approach both objectives by using reinforcement learning to compute the optimal control law. Unlike fixed exploration and exploitation balance, caution and probing are employed automatically by the controller in real-time, even after the learning process is terminated.
  arXiv  Detail & Related papers  (2023-09-18T18:05:35Z)
• End-to-End Intersection Handling using Multi-Agent Deep Reinforcement Learning [63.56464608571663]
  Navigating through intersections is one of the main challenging tasks for an autonomous vehicle. In this work, we focus on the implementation of a system able to navigate through intersections where only traffic signs are provided. We propose a multi-agent system using a continuous, model-free Deep Reinforcement Learning algorithm used to train a neural network for predicting both the acceleration and the steering angle at each time step.
  arXiv  Detail & Related papers  (2021-04-28T07:54:40Z)
• Congestion-aware Multi-agent Trajectory Prediction for Collision Avoidance [110.63037190641414]
  We propose to learn congestion patterns explicitly and devise a novel "Sense--Learn--Reason--Predict" framework. By decomposing the learning phases into two stages, a "student" can learn contextual cues from a "teacher" while generating collision-free trajectories. In experiments, we demonstrate that the proposed model is able to generate collision-free trajectory predictions in a synthetic dataset.
  arXiv  Detail & Related papers  (2021-03-26T02:42:33Z)
• Reinforcement Learning for Low-Thrust Trajectory Design of Interplanetary Missions [77.34726150561087]
  This paper investigates the use of reinforcement learning for the robust design of interplanetary trajectories in presence of severe disturbances. An open-source implementation of the state-of-the-art algorithm Proximal Policy Optimization is adopted. The resulting Guidance and Control Network provides both a robust nominal trajectory and the associated closed-loop guidance law.
  arXiv  Detail & Related papers  (2020-08-19T15:22:15Z)
• Model-Reference Reinforcement Learning for Collision-Free Tracking Control of Autonomous Surface Vehicles [1.7033108359337459]
  The proposed control algorithm combines a conventional control method with reinforcement learning to enhance control accuracy and intelligence. Thanks to reinforcement learning, the overall tracking controller is capable of compensating for model uncertainties and achieving collision avoidance.
  arXiv  Detail & Related papers  (2020-08-17T12:15:15Z)
• Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systems [81.7983463275447]
  Learning-based control algorithms require data collection with abundant supervision for training. We present a new approach for optimal motion planning with safe exploration that integrates chance-constrained optimal control with dynamics learning and feedback control.
  arXiv  Detail & Related papers  (2020-05-09T05:57:43Z)
• Learning Control Barrier Functions from Expert Demonstrations [69.23675822701357]
  We propose a learning based approach to safe controller synthesis based on control barrier functions (CBFs) We analyze an optimization-based approach to learning a CBF that enjoys provable safety guarantees under suitable Lipschitz assumptions on the underlying dynamical system. To the best of our knowledge, these are the first results that learn provably safe control barrier functions from data.
  arXiv  Detail & Related papers  (2020-04-07T12:29:06Z)
• Lane-Merging Using Policy-based Reinforcement Learning and Post-Optimization [0.0]
  We combine policy-based reinforcement learning with local optimization to foster and synthesize the best of the two methodologies. We evaluate the proposed method using lane-change scenarios with a varying number of vehicles.
  arXiv  Detail & Related papers  (2020-03-06T12:57:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.