Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

• URL: http://arxiv.org/abs/2303.17463v1
• Date: Thu, 30 Mar 2023 15:40:26 GMT
• Title: Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
• Authors: David Chapela-Campa, Ismail Benchekroun, Opher Baron, Marlon Dumas, Dmitry Krass, Arik Senderovich
• Abstract summary: Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. We propose a collection of measures to evaluate the quality of a BPS model.
• Score: 1.4027589547318842
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. For example, BPS allows us to estimate what would be the cycle time of a process if one or more resources became unavailable. The starting point of BPS is a process model annotated with simulation parameters (a BPS model). BPS models may be manually designed, based on information collected from stakeholders and empirical observations, or automatically discovered from execution data. Regardless of its origin, a key question when using a BPS model is how to assess its quality. In this paper, we propose a collection of measures to evaluate the quality of a BPS model w.r.t. its ability to replicate the observed behavior of the process. We advocate an approach whereby different measures tackle different process perspectives. We evaluate the ability of the proposed measures to discern the impact of modifications to a BPS model, and their ability to uncover the relative strengths and weaknesses of two approaches for automated discovery of BPS models. The evaluation shows that the measures not only capture how close a BPS model is to the observed behavior, but they also help us to identify sources of discrepancies.

Related papers

• MAP: Low-compute Model Merging with Amortized Pareto Fronts via Quadratic Approximation [80.47072100963017]
  Model merging is an effective approach to combine multiple single-task models, fine-tuned from the same pre-trained model, into a multitask model. Existing model-merging methods focus on enhancing average task accuracy. We introduce a novel low-compute algorithm, Model Merging with Amortized Pareto Front (MAP)
  arXiv  Detail & Related papers  (2024-06-11T17:55:25Z)
• White-box validation of quantitative product lines by statistical model checking and process mining [1.0918484507642576]
  We propose a novel methodology for validating software product line (PL) models by integrating Statistical Model Checking (SMC) with Process Mining (PM) Our approach focuses on the feature-oriented language QFLan in the PL engineering domain, allowing modeling of PLs with rich cross-tree and quantitative constraints.
  arXiv  Detail & Related papers  (2024-01-23T17:27:13Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Gaussian Process Probes (GPP) for Uncertainty-Aware Probing [61.91898698128994]
  We introduce a unified and simple framework for probing and measuring uncertainty about concepts represented by models. Our experiments show it can (1) probe a model's representations of concepts even with a very small number of examples, (2) accurately measure both epistemic uncertainty (how confident the probe is) and aleatory uncertainty (how fuzzy the concepts are to the model), and (3) detect out of distribution data using those uncertainty measures as well as classic methods do.
  arXiv  Detail & Related papers  (2023-05-29T17:00:16Z)
• Neural Enhanced Belief Propagation for Multiobject Tracking [8.228150100178983]
  We introduce a variant of BP that combines model-based with data-driven MOT. Our NEBP method improves tracking performance compared to model-based methods. We evaluate the performance of our NEBP approach for MOT on the nuScenes autonomous driving dataset.
  arXiv  Detail & Related papers  (2022-12-16T08:31:07Z)
• Enhancing Business Process Simulation Models with Extraneous Activity Delays [0.6073572808831218]
  This article proposes a method that discovers extraneous delays from event logs of business process executions. The proposed approach computes, for each pair of causally consecutive activity instances in the event log, the time when the target activity instance should theoretically have started. An empirical evaluation involving synthetic and real-life logs shows that the approach produces BPS models that better reflect the temporal dynamics of the process.
  arXiv  Detail & Related papers  (2022-06-28T14:51:10Z)
• XAI in the context of Predictive Process Monitoring: Too much to Reveal [3.10770247120758]
  Predictive Process Monitoring (PPM) has been integrated into process mining tools as a value-adding task. XAI methods are employed to compensate for the lack of transparency of most efficient predictive models. A comparison is missing to distinguish XAI characteristics or underlying conditions that are deterministic to an explanation.
  arXiv  Detail & Related papers  (2022-02-16T15:31:59Z)
• Proximal Reinforcement Learning: Efficient Off-Policy Evaluation in Partially Observed Markov Decision Processes [65.91730154730905]
  In applications of offline reinforcement learning to observational data, such as in healthcare or education, a general concern is that observed actions might be affected by unobserved factors. Here we tackle this by considering off-policy evaluation in a partially observed Markov decision process (POMDP) We extend the framework of proximal causal inference to our POMDP setting, providing a variety of settings where identification is made possible.
  arXiv  Detail & Related papers  (2021-10-28T17:46:14Z)
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models [95.8037674226622]
  We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
  arXiv  Detail & Related papers  (2021-02-17T18:25:30Z)
• Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion [59.549664231655726]
  A case-based reasoning (CBR) system solves a new problem by retrieving cases' that are similar to the given problem. In this paper, we demonstrate that such a system is achievable for reasoning in knowledge-bases (KBs) Our approach predicts attributes for an entity by gathering reasoning paths from similar entities in the KB.
  arXiv  Detail & Related papers  (2020-10-07T17:48:12Z)
• Structural Estimation of Partially Observable Markov Decision Processes [3.1614382994158956]
  We consider the structural estimation of the primitives of a POMDP model based upon the observable history of the process. We illustrate the estimation methodology with an application to optimal equipment replacement.
  arXiv  Detail & Related papers  (2020-08-02T15:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.