On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks

• URL: http://arxiv.org/abs/2504.21074v1
• Date: Tue, 29 Apr 2025 17:52:28 GMT
• Title: On the Potential of Large Language Models to Solve Semantics-Aware Process Mining Tasks
• Authors: Adrian Rebmann, Fabian David Schmidt, Goran Glavaš, Han van der Aa,
• Abstract summary: Large language models (LLMs) have shown to be valuable tools for tackling process mining tasks.<n>This paper systematically explores the capabilities of LLMs for semantics-aware process mining tasks.<n>We define five process mining tasks requiring semantic understanding and provide extensive benchmarking datasets for evaluation.
• Score: 3.9273545629281252
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have shown to be valuable tools for tackling process mining tasks. Existing studies report on their capability to support various data-driven process analyses and even, to some extent, that they are able to reason about how processes work. This reasoning ability suggests that there is potential for LLMs to tackle semantics-aware process mining tasks, which are tasks that rely on an understanding of the meaning of activities and their relationships. Examples of these include process discovery, where the meaning of activities can indicate their dependency, whereas in anomaly detection the meaning can be used to recognize process behavior that is abnormal. In this paper, we systematically explore the capabilities of LLMs for such tasks. Unlike prior work, which largely evaluates LLMs in their default state, we investigate their utility through both in-context learning and supervised fine-tuning. Concretely, we define five process mining tasks requiring semantic understanding and provide extensive benchmarking datasets for evaluation. Our experiments reveal that while LLMs struggle with challenging process mining tasks when used out of the box or with minimal in-context examples, they achieve strong performance when fine-tuned for these tasks across a broad range of process types and industries.

Related papers

• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Measuring Emergent Capabilities of LLMs for Software Engineering: How Far Are We? [45.126233498200534]
  We investigate the emergence of capabilities in the context of Software Engineering.<n>We propose a model-agnostic pipeline for evaluating this phenomenon across three SE tasks: bug fixing, code translation, and commit message generation.<n>Our findings do not provide evidence to support the idea of emergent capabilities resulting from scaling the model size in the selected set of tasks.
  arXiv  Detail & Related papers  (2024-11-26T22:48:55Z)
• ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure [0.0]
  We propose a benchmark that focuses on a specific aspect of reasoning ability: the direct evaluation of multi-step inference. Our dataset comprises pairs of explicit instructions and corresponding questions, where the procedures necessary for solving the questions are entirely detailed within the instructions. By constructing problems that require varying numbers of steps to solve and evaluating responses at each step, we enable a thorough assessment of state-of-the-art LLMs' ability to follow instructions.
  arXiv  Detail & Related papers  (2024-10-04T03:21:24Z)
• ProcessTBench: An LLM Plan Generation Dataset for Process Mining [0.0]
  Large Language Models (LLMs) have shown significant promise in plan generation. Existing datasets often lack the complexity needed for advanced tool use scenarios. We present the ProcessTBench synthetic dataset, an extension of the TaskBench dataset.
  arXiv  Detail & Related papers  (2024-09-13T20:56:21Z)
• Interpreting and Improving Large Language Models in Arithmetic Calculation [72.19753146621429]
  Large language models (LLMs) have demonstrated remarkable potential across numerous applications. In this work, we delve into uncovering a specific mechanism by which LLMs execute calculations. We investigate the potential benefits of selectively fine-tuning these essential heads/MLPs to boost the LLMs' computational performance.
  arXiv  Detail & Related papers  (2024-09-03T07:01:46Z)
• Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks [3.9273545629281252]
  Large language models (LLMs) could be used to tackle process mining tasks that benefit from an understanding of process behavior. In this paper, we investigate the capabilities of LLMs to tackle such semantics-aware process mining tasks.
  arXiv  Detail & Related papers  (2024-07-02T14:44:49Z)
• C-ICL: Contrastive In-context Learning for Information Extraction [54.39470114243744]
  c-ICL is a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods.
  arXiv  Detail & Related papers  (2024-02-17T11:28:08Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Tasks [54.71034943526973]
  In-context learning (ICL) has become the default method for using large language models (LLMs) We find that ICL falls short of handling specification-heavy tasks, which are tasks with complicated and extensive task specifications. We identify three primary reasons: inability to specifically understand context, misalignment in task schema comprehension with humans, and inadequate long-text understanding ability.
  arXiv  Detail & Related papers  (2023-11-15T14:26:30Z)
• Large Language Models can accomplish Business Process Management Tasks [0.0]
  We show how Large Language Models (LLMs) can accomplish text-related Business Process Management tasks. LLMs can accomplish process models from textual descriptions, mining declarative process models from textual descriptions, and assessing the suitability of process tasks from textual descriptions for robotic process automation.
  arXiv  Detail & Related papers  (2023-07-19T11:54:46Z)
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study [75.42182503265056]
  Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm. We deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks.
  arXiv  Detail & Related papers  (2021-05-08T22:26:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.