Rethinking World Models for Safety-Critical Embodied Systems

• URL: http://arxiv.org/abs/2609.03774v1
• Date: Thu, 03 Sep 2026 12:44:44 GMT
• Title: Rethinking World Models for Safety-Critical Embodied Systems
• Abstract summary: This perspective identifies three structural mismatches in current world modeling.<n>We propose the Risk-Informed World Model (RIWM) as a decision-centric research direction for safety-critical embodied systems.
• Score: 3.802367963338718
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World models have progressed from compact latent dynamics to generative, controllable, and interactive simulators of embodied environments. However, high predictive likelihood and visual fidelity do not necessarily ensure that a model preserves the evidence required for safe decision-making. This perspective identifies three structural mismatches in current world modeling: likelihood versus risk, prediction versus intervention, and finite-horizon prediction versus accumulated consequences. We propose the Risk-Informed World Model (RIWM) as a decision-centric research direction for safety-critical embodied systems. RIWM organizes world modeling around consequences, intervention, epistemic uncertainty, and recoverability, and integrates four interdependent capabilities: decision-relevant representation, counterfactual reasoning, safety-critical episodic memory, and runtime safety assurance. It distinguishes physical, social, and operational consequences while using epistemic uncertainty to qualify the evidence supporting action. We further discuss open challenges in identifying consequential futures, validating counterfactual reasoning, maintaining revisable safety memories, translating learned consequences into executable constraints, and determining when evidence is sufficient to act. This perspective argues that future world models should move beyond predicting likely futures toward identifying which futures matter, revising judgments through experience, and recognizing when to act, revise, sense, defer, or abstain.

Related papers

• World Models for Embodied Intelligence: From Plausible to Controllable to Actionable [100.13060753774657]
  World models connect perception and decision-making in embodied intelligence by maintaining hidden state, anticipating consequences, comparing interventions, and adapting when execution departs from expectations.<n>This raises a central question: which predictive capabilities improve behavior?<n>We introduce three progressively stronger capability levels: Plausible models preserve task-relevant temporal, geometric, or physical structure; Controllable models additionally predict how interventions alter that structure; and Actionable models translate predictions into measurable gains in planning, action, learning, evaluation, verification, recovery, or data selection.
  arXiv  Detail & Related papers  (2026-09-15T06:22:42Z)
• RiskWorld: Object-Centric Latent World Modeling for Autonomous Driving Risk Identification [30.32162964871242]
  RiskWorld is an object-centric latent world model that identifies risk from the imagined evolution of each candidate relative to the ego vehicle.<n>It decodes the rollout into object-level risk scores, supported by auxiliary future-relation and temporal-risk predictions.<n>On RiskBench, RiskWorld achieves the best overall F1 of 63.0% and the lowest false-alarm rate of 2.1%.
  arXiv  Detail & Related papers  (2026-08-12T02:52:04Z)
• The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems [26.594498540217568]
  In deployed systems, many of the most consequential failures are quieter than spectacular, distributed across components rather than localized in a single output, and normalized by ecosystem before they are recognized as hazards.<n>We argue that a central safety challenge in modern AI systems is not only whether a model emits harmful response, but whether the broader socio-technical system preserves the conditions under which errors remain visible, contestable, containable, and recoverable.
  arXiv  Detail & Related papers  (2026-07-21T17:02:37Z)
• Is the Future Compatible? Diagnosing Dynamic Consistency in World Action Models [28.165928090888986]
  World Action Models (WAMs) enable decision-making through imagined rollouts by predicting future observations and actions.<n>We identify action-state consistency, the alignment between predicted actions and induced state transitions, as a missing reliability axis for WAMs.<n>We introduce a value-free consensus strategy for test-time selection, which ranks candidate rollouts by agreement among predicted futures.
  arXiv  Detail & Related papers  (2026-05-08T09:44:43Z)
• Uncertainty-Aware Vision-based Risk Object Identification via Conformal Risk Tube Prediction [7.840136094141999]
  Existing approaches make decisions and ignore uncertainty, which could lead to safety-critical failures.<n>Despite these challenges, current methods lack principled framework to model risk uncertainty jointly across space and time.<n>We propose Conformal Risk Prediction unified formulation that captures principled risk uncertainty, provides coverage guarantees for true risks, and produces calibrated risk scores with uncertainty estimates.
  arXiv  Detail & Related papers  (2026-03-25T04:13:42Z)
• Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Driving [82.69496624372944]
  "Only driving like the expert" suffers from limited generalization.<n>Can an E2E-AD system make reliable decisions without any expert action supervision?<n>We propose a unified framework named Risk-aware World Model Predictive Control.
  arXiv  Detail & Related papers  (2026-02-26T17:32:30Z)
• Multimodal Belief-Space Covariance Steering with Active Probing and Influence for Interactive Driving [14.159772216001954]
  We introduce a hierarchical belief model that structures human behavior across coarse discrete intents and fine motion modes.<n>We develop an active probing strategy that identifies when multimodal ambiguity in human predictions may compromise safety.<n>Finally, a runtime risk-evaluation layer based on Conditional Value-at-Risk ensures that all probing actions remain within human risk tolerance during influence.
  arXiv  Detail & Related papers  (2026-02-16T08:04:16Z)
• From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models [77.04403907729738]
  This survey charts the evolution of uncertainty from a passive diagnostic metric to an active control signal guiding real-time model behavior.<n>We demonstrate how uncertainty is leveraged as an active control signal across three frontiers.<n>This survey argues that mastering the new trend of uncertainty is essential for building the next generation of scalable, reliable, and trustworthy AI.
  arXiv  Detail & Related papers  (2026-01-22T06:21:31Z)
• Towards provable probabilistic safety for scalable embodied AI systems [79.31011047593492]
  Embodied AI systems are increasingly prevalent across various applications.<n> Ensuring their safety in complex operating environments remains a major challenge.<n>This Perspective offers a pathway toward safer, large-scale adoption of embodied AI systems in safety-critical applications.
  arXiv  Detail & Related papers  (2025-06-05T15:46:25Z)
• Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework [54.40508478482667]
  We present a comprehensive framework to disentangle, quantify, and mitigate uncertainty in perception and plan generation.<n>We propose methods tailored to the unique properties of perception and decision-making.<n>We show that our uncertainty disentanglement framework reduces variability by up to 40% and enhances task success rates by 5% compared to baselines.
  arXiv  Detail & Related papers  (2024-11-03T17:32:00Z)
• Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction [55.569050872780224]
  We present an online framework for safe crowd-robot interaction based on risk-sensitive optimal control, wherein the risk is modeled by the entropic risk measure. Our modular approach decouples the crowd-robot interaction into learning-based prediction and model-based control. A simulation study and a real-world experiment show that the proposed framework can accomplish safe and efficient navigation while avoiding collisions with more than 50 humans in the scene.
  arXiv  Detail & Related papers  (2020-09-12T02:02:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.