Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

• URL: http://arxiv.org/abs/2606.11266v1
• Date: Tue, 09 Jun 2026 04:46:37 GMT
• Title: Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models
• Abstract summary: We present VLM-Safe-RL, a framework that integrates a frozen vision-language model into the CMDP Lagrangian update as an anticipatory cost term.<n>This is the first work to use frozen VLM signals as an anticipatory cost term inside the CMDP Lagrangian update.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The cost signal that constrained-RL algorithms optimize against is almost always reactive: the simulator emits a non-zero cost only after a collision has begun, and the Lagrange multiplier of PPO-Lagrangian grows only after the episode budget has been exceeded. At race speeds, where collisions are instantaneous and irreversible, any safety mechanism that waits for cost to accumulate is structurally too late. We present VLM-Safe-RL, a framework that integrates a frozen vision-language model into the CMDP Lagrangian update as an anticipatory cost term. The framework comprises four contributions: (i) Decoupled Dual-Path CLIP, independent reward/cost paths that respect the CMDP's factorization; (ii) VLM-Lagrange, an augmented multiplier update that incorporates a per-step VLM cost as an anticipatory term; (iii) Confidence Gating, a Bayes-optimal weight derived from a logistic noise model on the CLIP margin; and (iv) VLMPPOLag, the composed algorithm. On Safety-Gymnasium FormulaOne L2, our principal evaluation ($n{=}5$ seeds, $10^{6}$ steps, budget $d_{\text{lim}}{=}25$) VLMPPOLag$+$Conf is the only configuration in our default budget comparison that simultaneously retains substantive return ($J_r{\approx}40$) and holds cost within budget on a majority of seeds; the five constraint-aware baselines (PPOLag, CPO, CPPOPID, CPO-CLG, PPOLag-RND) each fail at least one requirement. The mechanism generalizes to held-out MetaDrive Medium (catastrophe rate $41\%{\to}26\%$, 95\% bootstrap CI $[-26,-5]$\,pp) and shows directionally consistent transfer to Bullet Safety-Gym; we report honestly where it does not (MetaDrive Easy/Hard, Qwen2-VL backbone) and trace the Hard failure to a Lagrangian-regulation pathology rather than the VLM signal itself. To our knowledge, this is the first work to use frozen VLM signals as an anticipatory cost term inside the CMDP Lagrangian update.

Related papers

• PCA: Persistence-Aware Compression and Aggregation for Fast Video Large Language Models [63.31550146180189]
  This paper introduces a training-free $mathbfP$ersistence-Aware $mathbfC$ompression and $mathbfA$ggregation (PCA) method to preserve high-sampling raw visual information before the encoding stage.<n>PCA consistently outperforms existing state-of-the-art approaches in both efficiency and accuracy.
  arXiv  Detail & Related papers  (2026-07-22T05:17:38Z)
• ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestration [8.226365534099399]
  We propose ZEBRA, a framework that reduces multi-phase budget allocation to a continuous nonlinear knapsack problem.<n>On a $150$-task APPS coding benchmark, both ZEBRA variants outperform LLM-direct on every aggregate metric.<n>We show that lightweight algorithmic guidance at inference time can improve the economic behavior of autonomous multi-agent systems.
  arXiv  Detail & Related papers  (2026-05-19T20:50:05Z)
• Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance [0.27998963147546146]
  We show that a single policy can learn control inputs and communication-efficient timing decisions under a pointwise Lyapunov safety shield.<n>A run-time assurance layer overrides the policy via a one-step-ahead Lyapunov prediction and a precomputed LQR backup.<n>A 12-state 3D quadrotor case study extends the framework to higher-dimensional systems where classical STC is intractable.
  arXiv  Detail & Related papers  (2026-05-11T23:55:15Z)
• How Many Iterations to Jailbreak? Dynamic Budget Allocation for Multi-Turn LLM Evaluation [22.523809021772802]
  Key events in large language models (LLMs) in multi-turn conversational settings often emerge only after repeated interactions.<n>Recent conformal survival frameworks construct reliable lower predictive bounds (LPBs) on the number of iterations to trigger the event of interest.<n>We introduce DAPRO, the first theoretically valid dynamic budget allocation framework for bounding the time-to-event in multi-turn interactions.
  arXiv  Detail & Related papers  (2026-05-07T17:25:15Z)
• Near-Optimal Primal-Dual Algorithm for Learning Linear Mixture CMDPs with Adversarial Rewards [0.8984888893275712]
  We study safe reinforcement learning in finite-horizon linear mixture constrained Markov decision processes.<n>We propose a primal-dual policy optimization algorithm that achieves regret and constraint violation bounds.<n>This is the first provably efficient algorithm for linear mixture CMDPs with adversarial rewards.
  arXiv  Detail & Related papers  (2026-03-29T21:51:33Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• Phase Transition for Budgeted Multi-Agent Synergy [41.486076708302456]
  Multi-agent systems can improve reliability, yet under a fixed inference budget they often help, saturate, or even collapse.<n>We develop a minimal and calibratable theory that predicts these regimes from three binding constraints of modern agent stacks.
  arXiv  Detail & Related papers  (2026-01-24T05:32:50Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• RL for Latent MDPs: Regret Guarantees and a Lower Bound [74.41782017817808]
  We consider the regret problem for reinforcement learning in latent Markov Decision Processes (LMDP) In an LMDP, an MDP is randomly drawn from a set of $M$ possible MDPs at the beginning of the interaction, but the identity of the chosen MDP is not revealed to the agent. We show that the key link is a notion of separation between the MDP system dynamics.
  arXiv  Detail & Related papers  (2021-02-09T16:49:58Z)
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization [105.7510838453122]
  We study the Safe Reinforcement Learning (SRL) problem using the Constrained Markov Decision Process (CMDP) formulation. We present an provably efficient online policy optimization algorithm for CMDP with safe exploration in the function approximation setting.
  arXiv  Detail & Related papers  (2020-03-01T17:47:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.