Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM

• URL: http://arxiv.org/abs/2607.28825v1
• Date: Thu, 30 Jul 2026 20:27:32 GMT
• Title: Building a Process-Modeling Tool using Agentic AI: An Experience Report on PM4Py-UCM
• Abstract summary: This paper reports on the AI-assisted construction of PM4Py-UCM, an open-source tool that mines Use Case Map (UCM) models from event logs.<n>We characterize the development record itself, composed of 18 agent sessions (374 human turns and 10,328 tool actions over 65 hours), 151 commits, 20 releases, and a test suite grown from 108 to 691 test functions.<n>Up to version 0.7.4, fixes outnumber features 2.3:1, with 18% of turns for correcting agent errors.
• Score: 1.3660182208844809
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enterprise-modeling (EM) tools are often complex and hard to extend. Yet, users may want to explore new EM features and capabilities that currently do not exist. AI coding agents can help here by enabling the development of new capabilities and entire tools, but whether we can trust a modeling-language tool an LLM largely wrote remains a question. This paper reports on the AI-assisted construction of PM4Py-UCM, an open-source tool that mines Use Case Map (UCM) models from event logs. PM4Py-UCM's capabilities include some expected from process mining tools (e.g., performance heat-maps and dashboards) and distinctive ones (e.g., mined executable scenarios/variants, and model decomposition). We mined the development record itself, composed of 18 agent sessions (374 human turns and 10,328 tool actions over 65 hours), 151 commits, 20 releases, and a test suite grown from 108 to 691 test functions, in order to characterize, in a single in-depth case, how the tool was built with an agent (Claude Code), complemented by an independent static assessment of the resulting code (coverage, complexity, maintainability, security, architecture). We contribute a reproducible, privacy-preserving toolkit and taxonomy that classify human turns and flag cross-cutting consistency work, agent corrections, and retracted requests. Up to version 0.7.4, fixes outnumber features 2.3:1, with ~18% of turns for correcting agent errors. Feature waves dragged a measurable tail of documentation/test/notebook consistency work, and tests grew lockstep with features. We finally present lessons learned, centered on making model transformations mechanically checkable, and the oracle-based validation strategy that closed the "the agent said it works" gap, for responsibly engineering EM tooling with AI.

Related papers

• Joint Optimization of Tool Creation and Use for Large Language Model Agents [69.07066297088187]
  SMITH (-grounded Multi-task Iterative Tool Honing) trains tool creation and tool use inside a single policy.<n>A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks.<n>It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline)
  arXiv  Detail & Related papers  (2026-08-25T13:59:34Z)
• The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior [27.33402415317721]
  We study tool architecture in coding agents through controlled experiments on repository-level issue fixing.<n>Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior.
  arXiv  Detail & Related papers  (2026-08-11T19:50:56Z)
• MADE: Belief-Driven Dual-Agent Coordination for Autonomous Model Deployment [19.126846876843356]
  LLM-based agents struggle with domain-specific tasks, motivating the integration of external tools.<n>Open-source community offers vast array of AI models typically released as heterogeneous research artifacts.<n>Transforming these into ready-to-call APIs is costly and labor-intensive.<n>We introduce Model Automated Deployment Engine (MADE), a dual-agent coordination system.
  arXiv  Detail & Related papers  (2026-08-02T12:14:43Z)
• When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning [58.75883713573783]
  Tool-Integrated Reasoning has emerged as a promising paradigm that incorporates tool call and execution within the reasoning trajectory.<n>We find that when the reasoning of the model conflicts with the tool results, the model tends to believe in its own reasoning.<n>We introduce Adaptive Tool Trust (ATTC), a framework that guides the model to adaptively choose to trust or ignore the tool results.
  arXiv  Detail & Related papers  (2026-04-09T14:14:37Z)
• OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents [49.34040731113563]
  We present OSWorld-MCP, the first comprehensive and fair benchmark for assessing computer-use agents' tool invocation, GUI operation, and decision-making abilities.<n> Rigorous manual validation yields 158 high-quality tools, each verified for correct functionality, practical applicability, and versatility.<n> OSWorld-MCP deepens understanding of multimodal agents and sets a new standard for evaluating performance in complex, tool-assisted environments.
  arXiv  Detail & Related papers  (2025-10-28T15:56:36Z)
• AlphaApollo: Orchestrating Foundation Models and Professional Tools into a Self-Evolving System for Deep Agentic Reasoning [110.57865233597762]
  We present AlphaApollo, a self-evolving agentic reasoning system.<n>It aims to address two bottlenecks in foundation model (FM) reasoning-limited model-intrinsic capacity and unreliable test-time computation.<n>AlphaApollo orchestrates multiple models with professional tools to enable deliberate, verifiable reasoning.
  arXiv  Detail & Related papers  (2025-10-05T15:42:24Z)
• MCP-Zero: Active Tool Discovery for Autonomous LLM Agents [13.005899769943442]
  We introduce MCP-Zero, an active agent framework that restores tool discovery autonomy to LLMs themselves.<n>Instead of overwhelming models with all available tools, MCP-Zero enables agents to actively identify capability gaps, and request specific tools on-demand.<n>We construct MCP-tools, a comprehensive dataset of 308 MCP servers and 2,797 tools from the official Model-Context-Protocol repository.
  arXiv  Detail & Related papers  (2025-06-01T15:48:53Z)
• LLM Agents Making Agent Tools [2.5529148902034637]
  Tool use has turned large language models (LLMs) into powerful agents that can perform complex multi-step tasks.<n>But these tools must be implemented in advance by human developers.<n>We propose ToolMaker, an agentic framework that autonomously transforms papers with code into LLM-compatible tools.
  arXiv  Detail & Related papers  (2025-02-17T11:44:11Z)
• SMART: Self-Aware Agent for Tool Overuse Mitigation [58.748554080273585]
  Current Large Language Model (LLM) agents demonstrate strong reasoning and tool use capabilities, but often lack self-awareness.<n>This imbalance leads to Tool Overuse, where models unnecessarily rely on external tools for tasks with parametric knowledge.<n>We introduce SMART (Strategic Model-Aware Reasoning with Tools), a paradigm that enhances an agent's self-awareness to optimize task handling and reduce tool overuse.
  arXiv  Detail & Related papers  (2025-02-17T04:50:37Z)
• Tool-Augmented Reward Modeling [58.381678612409]
  We propose a tool-augmented preference modeling approach, named Themis, to address limitations by empowering RMs with access to external environments. Our study delves into the integration of external tools into RMs, enabling them to interact with diverse external sources. In human evaluations, RLHF trained with Themis attains an average win rate of 32% when compared to baselines.
  arXiv  Detail & Related papers  (2023-10-02T09:47:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.