Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language

• URL: http://arxiv.org/abs/2604.19667v1
• Date: Tue, 21 Apr 2026 16:49:11 GMT
• Title: Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language
• Abstract summary: Chat2Workflow is a benchmark for generating executable visual directly from natural language.<n>We propose a robust agentic framework to mitigate recurrent execution errors.<n>Although our framework yields up to 5.34% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation.
• Score: 29.66870012781284
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write prompts for each step, and repeatedly revise the logic as requirements evolve-making development costly, time-consuming, and error-prone. To study whether large language models can automate this multi-round interaction process, we introduce Chat2Workflow, a benchmark for generating executable visual workflows directly from natural language, and propose a robust agentic framework to mitigate recurrent execution errors. Chat2Workflow is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially under complex or changing requirements. Although our agentic framework yields up to 5.34% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.

Related papers

• Natural-Language Workflows Are Not Software Yet: Artifact-Driven Compilation for Reliable Agent Execution [23.207195503032136]
  Artic is an artifact-driven workflow compiler that transforms a natural-language workflow into an artifact-driven workflow.<n>We show that compiled by Artic are 32 and 56 percentage points more consistent in cross-model and repeated-execution setups.
  arXiv  Detail & Related papers  (2026-08-21T17:47:47Z)
• From Search to Synthesis: Training LLMs as Zero-Shot Workflow Generators [17.263246226058474]
  Large language models (LLMs) excel across a wide range of tasks, yet their instance-specific solutions often lack the structural consistency needed for reliable deployment.<n>We present MetaFlow, which casts workflow generation as a meta-learning problem.<n>It trains in two stages: supervised fine-tuning on synthetic workflow data, followed by reinforcement learning with verifiable rewards.<n>The resulting model produces effective for trained tasks and exhibits strong generalization to untrained tasks and novel operator sets.
  arXiv  Detail & Related papers  (2026-06-29T14:17:09Z)
• From Specification to Execution: AI Assisted Scientific Workflow Management [0.7507031378170735]
  We present an AI-assisted approach to scientific workflow management.<n>The method introduces a structured specification phase that separates workflow intent, design, and implementation, allowing validation prior to code generation.<n>We evaluate the approach using a federated learning workflow for medical imaging, chosen for its parallel, iterative, and dependency-intensive structure.
  arXiv  Detail & Related papers  (2026-06-16T19:21:09Z)
• FlowMind: Execute-Summarize for Structured Workflow Generation from LLM Reasoning [5.153212048436295]
  LLMs can solve complex tasks through reasoning and tool use, but accurately translating these solutions into structured remains challenging.<n>We model as sequences of tool use and reformulate the problem as designing a mechanism that can both solve tasks and reliably construct them.<n>We propose an Execute-Summarize(ES) framework that decouples task execution from workflow construction.
  arXiv  Detail & Related papers  (2026-02-12T10:04:42Z)
• Learning to Compose for Cross-domain Agentic Workflow Generation [56.630382886594184]
  We create an open-source LLM for cross-domain workflow generation.<n>We learn a compact set of reusable workflow capabilities across diverse domains.<n>Our 1-pass generator surpasses SOTA refinement baselines that consume 20 iterations.
  arXiv  Detail & Related papers  (2026-02-11T18:27:22Z)
• DyFlow: Dynamic Workflow Framework for Agentic Reasoning [79.19799197382478]
  DyFlow is a dynamic workflow generation framework that adaptively constructs and adjusts reasoning procedures based on task requirements and real-time intermediate feedback.<n>We systematically evaluate DyFlow across diverse domains, including social reasoning, biomedical tasks, mathematical problem solving, and code generation.<n>Results demonstrate that DyFlow significantly outperforms existing baselines, achieving substantial Pass@k improvements and exhibiting robust generalization across diverse domains.
  arXiv  Detail & Related papers  (2025-09-30T10:36:23Z)
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym [52.31172214690965]
  We introduce CodeGym, a framework that synthesizes diverse, verifiable, and controllable multi-turn tool-use environments for agent RL.<n>CodeGym rewrites static coding problems into interactive environments by extracting atomic functions or logic into callable tools.<n>Models of varying sizes and chain-of-thought configurations, trained in CodeGym, exhibit consistent out-of-distribution generalizability.
  arXiv  Detail & Related papers  (2025-09-22T03:03:56Z)
• WorkTeam: Constructing Workflows from Natural Language with Multi-Agents [6.656951366751657]
  Hand-crafted workflow construction requires expert knowledge, presenting significant technical barriers.<n>We propose WorkTeam, a multi-agent NL2Workflow framework comprising a supervisor, orchestrator, and filler agent.<n>Our approach significantly increases the success rate of workflow construction, providing a novel and effective solution for enterprise NL2Workflow services.
  arXiv  Detail & Related papers  (2025-03-28T14:33:29Z)
• Flow: Modularized Agentic Workflow Automation [53.073598156915615]
  Multi-agent frameworks powered by large language models (LLMs) have demonstrated great success in automated planning and task execution.<n>However, the effective adjustment of agentic during execution has not been well studied.<n>In this paper, we define an activity-on-vertex (AOV) graph, which allows continuous workflow refinement by agents.<n>Our proposed multi-agent framework achieves efficient concurrent execution of subtasks, effective goal achievement, and enhanced error tolerance.
  arXiv  Detail & Related papers  (2025-01-14T04:35:37Z)
• Opus: A Large Work Model for Complex Workflow Generation [0.0]
  Opus is a framework for generating and optimizing tasks tailored to complex Business Process Outsourcing (BPO) use cases.<n>Our approach generates executables from Intention, defined as the alignment of Client Input, Client Output and Process Directed Context.
  arXiv  Detail & Related papers  (2024-11-30T20:00:41Z)
• WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models [105.46456444315693]
  We presentLLM, a data-centric framework to enhance the capability of large language models in workflow orchestration. It first constructs a large-scale fine-tuningBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. LlamaLlama demonstrates a strong capacity to orchestrate complex APIs, while also achieving notable generalization performance.
  arXiv  Detail & Related papers  (2024-11-08T09:58:02Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• AutoFlow: Automated Workflow Generation for Large Language Model Agents [39.72700864347576]
  Large Language Models (LLMs) have shown significant progress in understanding complex natural language. To make sure LLM Agents follow an effective and reliable procedure to solve the given task, manually designed are usually used. We propose AutoFlow, a framework designed to automatically generate for agents to solve complex tasks.
  arXiv  Detail & Related papers  (2024-07-01T21:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.