SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model

• URL: http://arxiv.org/abs/2606.20698v1
• Date: Mon, 15 Jun 2026 08:58:37 GMT
• Title: SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Model
• Abstract summary: We propose SafeDojo, a model-based safe reinforcement learning framework for vision-language-action policies.<n>Online reinforcement learning is performed on top of an interactive video world model.<n>On SafeLIBERO, SafeDojo achieves the best aggregate task success, safe success, and execution efficiency.
• Score: 45.895712358791705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safe control is a prerequisite for real-world embodied intelligence, for which safe reinforcement learning has emerged as a promising paradigm. However, existing safe reinforcement learning methods either require costly real-world exploration or depend on hand-crafted safety functions. Neither scales to vision-language-action models deployed in open-world physical environments. We propose SafeDojo, the first model-based safe reinforcement learning framework for vision-language-action policies designed to learn safe actions through world model-based imagination. Specifically, SafeDojo performs online reinforcement learning on top of an interactive video world model. The world model generates action-conditioned future predictions, from which a tailored ResNet success classifier estimates per-step task progress from imagined frames and a lightweight safety head predicts per-step safety costs from latent context together with the proposed action chunk, enabling simultaneous assessment of task execution and trajectory safety. The decoupled task-reward and safety-cost signals are balanced through a Lagrangian-based constrained GRPO objective, enabling coordinated improvement of task success and safety under explicit constraints. On SafeLIBERO, SafeDojo achieves the best aggregate task success, safe success, and execution efficiency among inference-time safety, model-free RL, and model-based RL baselines, with the best average safe-success rate on both levels and an 8.25 percentage-point improvement over the strongest baseline on Level I. Real-world Franka deployment further shows the best average task and safe-success rates across five tasks. Our results position world model-based safe reinforcement learning as a scalable and generalizable path toward safe embodied intelligence.

Related papers

• Yuvion LLM: An Adversarially-Aware Large Language Model for Content And AI Safety [67.86055044301408]
  We present Yuvion LLM, a large language model built for adversarially robust content safety and broader AI safety.<n>Its pipeline combines adversarially aware data construction, knowledge-enhanced continued pretraining, and policy-grounded multi-task safety post-training.<n>Yuvion LLM demonstrates clear advantages on safety-focused benchmarks and particularly strong robustness under adversarial conditions.
  arXiv  Detail & Related papers  (2026-06-26T01:12:02Z)
• SafeWork-R1: Coevolving Safety and Intelligence under the AI-45$^{\circ}$ Law [91.33824439029533]
  We introduce SafeWork-R1, a cutting-edge multimodal reasoning model that demonstrates the coevolution of capabilities and safety.<n>It is developed by our proposed SafeLadder framework, which incorporates large-scale, progressive, safety-oriented reinforcement learning post-training.<n>We further develop SafeWork-R1-InternVL3-78B, SafeWork-R1-DeepSeek-70B, and SafeWork-R1-Qwen2.5VL-7B.
  arXiv  Detail & Related papers  (2025-07-24T16:49:19Z)
• VL-SAFE: Vision-Language Guided Safety-Aware Reinforcement Learning with World Models for Autonomous Driving [1.9242820889313577]
  Reinforcement learning (RL)-based autonomous driving policy learning faces critical limitations.<n>RL often fail to capture the true semantic meaning of "safety" in complex driving contexts.<n>We propose VL-SAFE, a world model-based safe RL framework with Vision-Language model (VLM)-as-safety-guidance paradigm.
  arXiv  Detail & Related papers  (2025-05-22T08:29:59Z)
• SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning [30.037309138373754]
  Vision-language-action models (VLAs) show potential as generalist robot policies.<n>These models pose extreme safety challenges during real-world deployment, including the risk of harm to the environment, the robot itself, and humans.<n>We address this by exploring an integrated safety approach (ISA), systematically modeling safety requirements, then actively eliciting diverse unsafe behaviors.
  arXiv  Detail & Related papers  (2025-03-05T13:16:55Z)
• Progressive Safeguards for Safe and Model-Agnostic Reinforcement Learning [5.593642806259113]
  We model a meta-learning process where each task is synchronized with a safeguard that monitors safety and provides a reward signal to the agent. The design of the safeguard is manual but it is high-level and model-agnostic, which gives rise to an end-to-end safe learning approach. We evaluate our framework in a Minecraft-inspired Gridworld, a VizDoom game environment, and an LLM fine-tuning application.
  arXiv  Detail & Related papers  (2024-10-31T16:28:33Z)
• ActSafe: Active Exploration with Safety Constraints for Reinforcement Learning [48.536695794883826]
  We present ActSafe, a novel model-based RL algorithm for safe and efficient exploration.<n>We show that ActSafe guarantees safety during learning while also obtaining a near-optimal policy in finite time.<n>In addition, we propose a practical variant of ActSafe that builds on latest model-based RL advancements.
  arXiv  Detail & Related papers  (2024-10-12T10:46:02Z)
• SafeDreamer: Safe Reinforcement Learning with World Models [7.773096110271637]
  We introduce SafeDreamer, a novel algorithm incorporating Lagrangian-based methods into world model planning processes. Our method achieves nearly zero-cost performance on various tasks, spanning low-dimensional and vision-only input.
  arXiv  Detail & Related papers  (2023-07-14T06:00:08Z)
• Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning [64.11013095004786]
  We propose a dual-agent safe reinforcement learning strategy consisting of a baseline and a safe agent. Such a decoupled framework enables high flexibility, data efficiency and risk-awareness for RL-based control. The proposed method outperforms the state-of-the-art safe RL algorithms on difficult robot locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2022-12-14T03:11:25Z)
• SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition [59.94644674087599]
  We propose SAFEty skill pRiors (SAFER), an algorithm that accelerates policy learning on complex control tasks under safety constraints. Through principled training on an offline dataset, SAFER learns to extract safe primitive skills. In the inference stage, policies trained with SAFER learn to compose safe skills into successful policies.
  arXiv  Detail & Related papers  (2022-02-10T05:43:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.