SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

• URL: http://arxiv.org/abs/2607.15550v1
• Date: Fri, 17 Jul 2026 01:45:54 GMT
• Title: SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction
• Abstract summary: SeerGuard is a consequence-aware safety framework designed to mitigate risks through pre-execution instruction-level screening and action-level risk assessment.<n>We construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment.
• Score: 26.773000505309437
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks where a single erroneous action can lead to irreversible consequences. Existing safety mechanisms are primarily reactive, lacking the ability to assess risks before execution. In this paper, we introduce SeerGuard, a consequence-aware safety framework designed to mitigate these risks through pre-execution instruction-level screening and action-level risk assessment. Specifically, the action-level assessment analyzes agent-proposed actions within current GUI states, anticipating likely outcomes to identify risks before they are executed. To enable these capabilities, we construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents. On Qwen3-VL-8B-Instruct, it increases the safety-utility score from $0.191$ to $0.596$ at $ω=0.8$ and reduces the risk-cost score from $0.347$ to $0.130$ at $α=0.8$. Further analyses on our SAWM validate the effectiveness of the instruction-level screening, alongside the capability of action risk assessment and next-state prediction.

Related papers

• AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories [12.694014584212695]
  We introduce AURA-Eval, a framework combining controlled augmentation with granular diagnosis of behavior in tool-use trajectories.<n>Using 157 sourced trajectories, we generate 1,249 evaluation items and evaluate 20 frontier and open-weight models.<n>Our results show that LLM agents engage in unsafe behavior more often when no safe path exists.
  arXiv  Detail & Related papers  (2026-09-06T18:58:56Z)
• MobileWorldSafety: Benchmarking GUI Agent Safety Against Environmental Injection Attacks in Android Apps [26.50307679295416]
  We introduce MobileWorldSafety, a benchmark of risk tasks built on real Android applications.<n>For each task, we define a verifiable risk indicator over the final system state.<n>This distinguishes safety failures from capability failures and enables objective and reproducible assessment.
  arXiv  Detail & Related papers  (2026-08-18T11:33:22Z)
• VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring [60.53395558502203]
  Vision-Language Embodied Safety Agent (VLESA) monitors human activities from egocentric video.<n>VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context.<n>An intent-action prediction agent is proposed to jointly infer goals and predict future actions from video.
  arXiv  Detail & Related papers  (2026-06-02T17:42:17Z)
• CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation [68.53387633351484]
  We propose CORA (COnformal Risk-controlled GUI Agent), a post-policy, pre-action safeguarding framework that provides statistical guarantees on harmful executed actions.<n> CORA reformulates safety as selective action execution: we train a Guardian model to estimate action-conditional risk for each proposed step.<n>To rigorously evaluate this paradigm, we introduce Phone-Harm, a new benchmark of mobile safety violations with step-level harm labels.
  arXiv  Detail & Related papers  (2026-04-10T09:41:21Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• SafeRBench: A Comprehensive Benchmark for Safety Assessment in Large Reasoning Models [60.8821834954637]
  We present SafeRBench, the first benchmark that assesses LRM safety end-to-end.<n>We pioneer the incorporation of risk categories and levels into input design.<n>We introduce a micro-thought chunking mechanism to segment long reasoning traces into semantically coherent units.
  arXiv  Detail & Related papers  (2025-11-19T06:46:33Z)
• DeepKnown-Guard: A Proprietary Model-Based Safety Response Framework for AI Agents [12.054307827384415]
  Large Language Models (LLMs) have become increasingly prominent, severely constraining their trustworthy deployment in critical domains.<n>This paper proposes a novel safety response framework designed to safeguard LLMs at both the input and output levels.
  arXiv  Detail & Related papers  (2025-11-05T03:04:35Z)
• LM Agents May Fail to Act on Their Own Risk Knowledge [15.60032437959883]
  Language model (LM) agents pose a diverse array of potential, severe risks in safety-critical scenarios.<n>While they often answer "Yes" to queries like "Is executing sudo rm -rf /*' dangerous?", they will likely fail to identify such risks in instantiated trajectories.
  arXiv  Detail & Related papers  (2025-08-19T02:46:08Z)
• AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions [64.85086226439954]
  We present SAFE, a benchmark for assessing the safety of embodied VLM agents on hazardous instructions.<n> SAFE comprises three components: SAFE-THOR, SAFE-VERSE, and SAFE-DIAGNOSE.<n>We uncover systematic failures in translating hazard recognition into safe planning and execution.
  arXiv  Detail & Related papers  (2025-06-17T16:37:35Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.