World of Workflows: a Benchmark for Bringing World Models to Enterprise Systems

• URL: http://arxiv.org/abs/2601.22130v1
• Date: Thu, 29 Jan 2026 18:51:54 GMT
• Title: World of Workflows: a Benchmark for Bringing World Models to Enterprise Systems
• Abstract summary: Frontier large language models (LLMs) excel in many domains, yet they remain untested in complex enterprise systems.<n>We introduce World of Warcraft (WoW), a realistic ServiceNow-based environment incorporating 4,000+ business rules and 55 active embedded in the system.<n>We reveal two major takeaways: (1) Frontier LLMs suffer from dynamics blindness, consistently failing to predict the invisible, cascading side effects of their actions, which leads to silent constraint violations, and (2) reliability in opaque systems requires grounded world modeling, where agents must mentally simulate hidden state transitions to bridge the observability gap when high-fidelity feedback is unavailable.
• Score: 6.643798239990755
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier large language models (LLMs) excel as autonomous agents in many domains, yet they remain untested in complex enterprise systems where hidden workflows create cascading effects across interconnected databases. Existing enterprise benchmarks evaluate surface-level agentic task completion similar to general consumer benchmarks, ignoring true challenges in enterprises, such as limited observability, large database state, and hidden workflows with cascading side effects. We introduce World of Workflows (WoW), a realistic ServiceNow-based environment incorporating 4,000+ business rules and 55 active workflows embedded in the system, alongside WoW-bench, a benchmark of 234 tasks evaluating constrained agentic task completion and enterprise dynamics modeling capabilities. We reveal two major takeaways: (1) Frontier LLMs suffer from dynamics blindness, consistently failing to predict the invisible, cascading side effects of their actions, which leads to silent constraint violations, and (2) reliability in opaque systems requires grounded world modeling, where agents must mentally simulate hidden state transitions to bridge the observability gap when high-fidelity feedback is unavailable. For reliable and useful enterprise agents, WoW motivates a new paradigm to explicitly learn system dynamics. We release our GitHub for setting up and evaluating WoW.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks [98.07146747892239]
  OSWorld 2.0 is a benchmark of 108 long-horizon computer-use across everyday and professional tasks.<n>Each task represents a realistic end-to-end workflow that takes human users a median of about 1.6 hours to complete.
  arXiv  Detail & Related papers  (2026-06-28T17:59:17Z)
• Running the Gauntlet: Re-evaluating the Capabilities of Agents Beyond Familiar Environments [67.2234316079859]
  GauntletBench is a web-based benchmark for evaluating agent generalisation in challenging scenarios.<n>It focuses on three underexplored capabilities (temporal perception, graphical understanding, and 3D reasoning) across five less-covered professional applications.<n>Our empirical results reveal that frontier agentic systems remain far from achieving human-level performance.
  arXiv  Detail & Related papers  (2026-06-12T12:32:24Z)
• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions [78.49000936275773]
  We introduce AgentHijack, a benchmark designed to evaluate the robustness of computer-use agents under common corruptions.<n>We evaluate a variety of desktop tasks that utilize MLLM-based agents and discover that even minor instances of corruption can result in substantial performance degradation.<n>We propose AgentHijack-Agent, a framework that integrates an action generator with enhanced grounding capabilities and an onlooker responsible for behavior summarization and environment checking.
  arXiv  Detail & Related papers  (2026-05-25T11:09:22Z)
• Do Enterprise Systems Need Learned World Models? The Importance of Context to Infer Dynamics [16.09809323840653]
  We show that offline-trained world models can perform well in-distribution but degrade as dynamics change, whereas discovery-based agents are more robust under shift.<n>Our findings suggest that, in enterprise environments, agents should not rely solely on fixed internalized dynamics, but should incorporate mechanisms for discovering relevant transition logic at runtime.
  arXiv  Detail & Related papers  (2026-05-12T14:22:10Z)
• TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualization [4.62716665682001]
  TopoPilot is a reliable and agentic framework for automating complex scientific visualization.<n>TopoPilot incorporates systematic guardrails and verification mechanisms to ensure reliable operation.<n>In evaluations, TopoPilot achieves a success rate exceeding 99%, compared to under 50% for baselines without comprehensive guardrails and checks.
  arXiv  Detail & Related papers  (2026-03-26T05:56:53Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• Computer-Using World Model [58.59112582915026]
  We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next user interface (UI) state.<n> CUWM first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot.<n>We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution.
  arXiv  Detail & Related papers  (2026-02-19T13:48:29Z)
• EntWorld: A Holistic Environment and Benchmark for Verifiable Enterprise GUI Agents [12.7922877987936]
  EntWorld is a large-scale benchmark consisting of 1,756 tasks across six representative enterprise domains.<n>We propose a schema-grounded task generation framework that directly reverse-engineers business logic from underlying database schemas.<n>We show that state-of-the-art models achieve 47.61% success rate on EntWorld, substantially lower than the human performance.
  arXiv  Detail & Related papers  (2026-01-25T06:58:15Z)
• What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding [50.35012849818872]
  Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks.<n>We propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding.<n>Our experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment.
  arXiv  Detail & Related papers  (2026-01-14T14:09:11Z)
• REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation [46.970355660265284]
  Vision-Language-Action (VLA) models empower robots to understand and execute tasks described by natural language instructions.<n>A key challenge lies in their ability to generalize beyond the specific environments and conditions they were trained on.<n>We present REALM, a new simulation environment and benchmark designed to evaluate the generalization capabilities of VLA models.
  arXiv  Detail & Related papers  (2025-12-22T16:44:23Z)
• IndustryNav: Exploring Spatial Reasoning of Embodied Agents in Dynamic Industrial Navigation [56.43007596544299]
  IndustryNav is the first dynamic industrial navigation benchmark for active spatial reasoning.<n>A study of nine state-of-the-art Visual Large Language Models reveals that closed-source models maintain a consistent advantage.
  arXiv  Detail & Related papers  (2025-11-21T16:48:49Z)
• WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model [55.276852838877346]
  Self-evolving agents are trained on trajectories sampled autonomously based on their own policies.<n>We propose a novel framework that introduces a co-evolving World Model LLM.<n>This world model predicts the next observation based on the current observation and action within the web environment.
  arXiv  Detail & Related papers  (2025-04-23T02:54:31Z)
• AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? [50.36826943689364]
  We study whether language agents can perform realistic and time-consuming tasks on the web. We introduce AssistantBench, a new benchmark consisting of 214 realistic tasks that can be automatically evaluated. We find that AssistantBench exposes the limitations of current systems, including language models and retrieval-augmented language models.
  arXiv  Detail & Related papers  (2024-07-22T15:18:45Z)
• LEGENT: Open Platform for Embodied Agents [60.71847900126832]
  We introduce LEGENT, an open, scalable platform for developing embodied agents using Large Language Models (LLMs) and Large Multimodal Models (LMMs) LEGENT offers a rich, interactive 3D environment with communicable and actionable agents, paired with a user-friendly interface. In experiments, an embryonic vision-language-action model trained on LEGENT-generated data surpasses GPT-4V in embodied tasks.
  arXiv  Detail & Related papers  (2024-04-28T16:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.