Do Multimodal Foundation Models Understand Enterprise Workflows? A Benchmark for Business Process Management Tasks

• URL: http://arxiv.org/abs/2406.13264v1
• Date: Wed, 19 Jun 2024 06:50:15 GMT
• Title: Do Multimodal Foundation Models Understand Enterprise Workflows? A Benchmark for Business Process Management Tasks
• Authors: Michael Wornow, Avanika Narayan, Ben Viggiano, Ishan S. Khare, Tathagat Verma, Tibor Thompson, Miguel Angel Fuentes Hernandez, Sudharsan Sundar, Chloe Trujillo, Krrish Chawla, Rongfei Lu, Justin Shen, Divya Nagaraj, Joshua Martinez, Vardhan Agrawal, Althea Hudson, Nigam H. Shah, Christopher Re,
• Abstract summary: Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. Our benchmark shows that while state-of-the-art FMs can automatically generate documentation, they struggle to re-apply that knowledge towards finer-grained validation of workflow completion.
• Score: 11.701910903349201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing ML benchmarks lack the depth and diversity of annotations needed for evaluating models on business process management (BPM) tasks. BPM is the practice of documenting, measuring, improving, and automating enterprise workflows. However, research has focused almost exclusively on one task - full end-to-end automation using agents based on multimodal foundation models (FMs) like GPT-4. This focus on automation ignores the reality of how most BPM tools are applied today - simply documenting the relevant workflow takes 60% of the time of the typical process optimization project. To address this gap we present WONDERBREAD, the first benchmark for evaluating multimodal FMs on BPM tasks beyond automation. Our contributions are: (1) a dataset containing 2928 documented workflow demonstrations; (2) 6 novel BPM tasks sourced from real-world applications ranging from workflow documentation to knowledge transfer to process improvement; and (3) an automated evaluation harness. Our benchmark shows that while state-of-the-art FMs can automatically generate documentation (e.g. recalling 88% of the steps taken in a video demonstration of a workflow), they struggle to re-apply that knowledge towards finer-grained validation of workflow completion (F1 < 0.3). We hope WONDERBREAD encourages the development of more "human-centered" AI tooling for enterprise applications and furthers the exploration of multimodal FMs for the broader universe of BPM tasks. We publish our dataset and experiments here: https://github.com/HazyResearch/wonderbread

Related papers

• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [73.81908518992161]
  We introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering. Spider2-V features real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems.
  arXiv  Detail & Related papers  (2024-07-15T17:54:37Z)
• Machine learning in business process management: A systematic literature review [0.0]
  Machine learning (ML) provides algorithms to create computer programs based on data without explicitly programming them. Three frequent examples of using ML are providing decision support through predictions, discovering accurate process models, and improving resource allocation. This study is the first exhaustive review of how ML has been used in BPM.
  arXiv  Detail & Related papers  (2024-05-26T01:12:24Z)
• Automating the Enterprise with Foundation Models [15.708380634503467]
  ECLAIR is a system to automate enterprise with minimal human supervision. We identify human-AI collaboration, validation, and self-improvement as open challenges.
  arXiv  Detail & Related papers  (2024-05-03T23:25:15Z)
• FlowMind: Automatic Workflow Generation with LLMs [12.848562107014093]
  This paper introduces a novel approach, FlowMind, leveraging the capabilities of Large Language Models (LLMs) We propose a generic prompt recipe for a lecture that helps ground LLM reasoning with reliable Application Programming Interfaces (APIs) We also introduce NCEN-QA, a new dataset in finance for benchmarking question-answering tasks from N-CEN reports on funds.
  arXiv  Detail & Related papers  (2024-03-17T00:36:37Z)
• The Foundations of Computational Management: A Systematic Approach to Task Automation for the Integration of Artificial Intelligence into Existing Workflows [55.2480439325792]
  This article introduces Computational Management, a systematic approach to task automation. The article offers three easy step-by-step procedures to begin the process of implementing AI within a workflow.
  arXiv  Detail & Related papers  (2024-02-07T01:45:14Z)
• Accelerated Cloud for Artificial Intelligence (ACAI) [24.40451195277244]
  We propose an end-to-end cloud-based machine learning platform, Accelerated Cloud for AI (ACAI) ACAI enables cloud-based storage of indexed, labeled, and searchable data, as well as automatic resource provisioning, job scheduling, and experiment tracking. We show that our auto-provisioner produces a 1.7x speed-up and 39% cost reduction, and our system reduces experiment time for ML scientists by 20% on typical ML use cases.
  arXiv  Detail & Related papers  (2024-01-30T07:09:48Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [85.3879908356586]
  We introduce TaskBench to evaluate the capability of large language models in task automation. To generate high-quality evaluation datasets, we introduce the concept of Tool Graph. We also propose TaskEval to evaluate the capability of LLMs from different aspects, including task decomposition, tool invocation, and parameter prediction.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks. This paper presents the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)
• Just Tell Me: Prompt Engineering in Business Process Management [63.08166397142146]
  GPT-3 and other language models (LMs) can effectively address various natural language processing (NLP) tasks. We argue that prompt engineering can help bring the capabilities of LMs to BPM research.
  arXiv  Detail & Related papers  (2023-04-14T14:55:19Z)
• FETA: Towards Specializing Foundation Models for Expert Task Applications [49.57393504125937]
  Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. We show in this paper that FMs still have poor out-of-the-box performance on expert tasks. We propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation.
  arXiv  Detail & Related papers  (2022-09-08T08:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.