Process Extraction from Text: Benchmarking the State of the Art and Paving the Way for Future Challenges

• URL: http://arxiv.org/abs/2110.03754v2
• Date: Wed, 25 Oct 2023 11:11:16 GMT
• Title: Process Extraction from Text: Benchmarking the State of the Art and Paving the Way for Future Challenges
• Authors: Patrizio Bellan, Mauro Dragoni, Chiara Ghidini, Han van der Aa, Simone Paolo Ponzetto
• Abstract summary: It is unclear how well existing solutions are able to solve the model-extraction problem and how they compare to each other. We compare 10 state-of-the-art approaches for model extraction in a systematic manner, covering both qualitative and quantitative aspects. The results show three distinct groups of tools in terms of performance, with no tool obtaining very good scores and also serious limitations.
• Score: 18.485565445940175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The extraction of process models from text refers to the problem of turning the information contained in an unstructured textual process descriptions into a formal representation,i.e.,a process model. Several automated approaches have been proposed to tackle this problem, but they are highly heterogeneous in scope and underlying assumptions,i.e., differences in input, target output, and data used in their evaluation.As a result, it is currently unclear how well existing solutions are able to solve the model-extraction problem and how they compare to each other.We overcome this issue by comparing 10 state-of-the-art approaches for model extraction in a systematic manner, covering both qualitative and quantitative aspects.The qualitative evaluation compares the analysis of the primary studies on: 1 the main characteristics of each solution;2 the type of process model elements extracted from the input data;3 the experimental evaluation performed to evaluate the proposed framework.The results show a heterogeneity of techniques, elements extracted and evaluations conducted, that are often impossible to compare.To overcome this difficulty we propose a quantitative comparison of the tools proposed by the papers on the unifying task of process model entity and relation extraction so as to be able to compare them directly.The results show three distinct groups of tools in terms of performance, with no tool obtaining very good scores and also serious limitations.Moreover, the proposed evaluation pipeline can be considered a reference task on a well-defined dataset and metrics that can be used to compare new tools. The paper also presents a reflection on the results of the qualitative and quantitative evaluation on the limitations and challenges that the community needs to address in the future to produce significant advances in this area.

Related papers

• An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models [2.1945750784330067]
  This research evaluates summarization performance across 17 large language models (OpenAI, Google, Anthropic, open-source) We assessed models on seven diverse datasets using metrics for factual consistency, semantic similarity, lexical overlap, and human-like quality.
  arXiv  Detail & Related papers  (2025-04-06T16:24:22Z)
• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Deep Model Interpretation with Limited Data : A Coreset-based Approach [0.810304644344495]
  We propose a coreset-based interpretation framework that utilizes coreset selection methods to sample a representative subset of the large dataset for the interpretation task. We propose a similarity-based evaluation protocol to assess the robustness of model interpretation methods towards the amount data they take as input.
  arXiv  Detail & Related papers  (2024-10-01T09:07:24Z)
• Pattern based learning and optimisation through pricing for bin packing problem [50.83768979636913]
  We argue that when problem conditions such as the distributions of random variables change, the patterns that performed well in previous circumstances may become less effective. We propose a novel scheme to efficiently identify patterns and dynamically quantify their values for each specific condition. Our method quantifies the value of patterns based on their ability to satisfy constraints and their effects on the objective value.
  arXiv  Detail & Related papers  (2024-08-27T17:03:48Z)
• Multi-objective Binary Differential Approach with Parameter Tuning for Discovering Business Process Models: MoD-ProM [2.3423913554158653]
  We consider the Binary Differential Evolution approach in a multi-objective framework for the task of process discovery. It is shown that the process models generated by the proposed approach are superior to or at least as good as those generated by the state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-06-25T16:53:55Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models [22.522156479335706]
  This paper conducts the first large-scale empirical study to compare the performance of existing state-of-the-art fairness improving techniques. Our findings reveal substantial variations in the performance of each method across different datasets and sensitive attributes. Different fairness evaluation metrics, due to their distinct focuses, yield significantly different assessment results.
  arXiv  Detail & Related papers  (2024-01-08T06:53:33Z)
• How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey [23.757740341834126]
  We show that H-Score generally performs well with superiorities in effectiveness and efficiency. We also outline the difficulties of consideration of training details, applicability to text generation, and consistency to certain metrics which shed light on future directions.
  arXiv  Detail & Related papers  (2023-12-08T01:17:28Z)
• Better Understanding Differences in Attribution Methods via Systematic Evaluations [57.35035463793008]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models.
  arXiv  Detail & Related papers  (2023-03-21T14:24:58Z)
• Operationalizing Specifications, In Addition to Test Sets for Evaluating Constrained Generative Models [17.914521288548844]
  We argue that the scale of generative models could be exploited to raise the abstraction level at which evaluation itself is conducted. Our recommendations are based on leveraging specifications as a powerful instrument to evaluate generation quality.
  arXiv  Detail & Related papers  (2022-11-19T06:39:43Z)
• Towards Better Understanding Attribution Methods [77.1487219861185]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We also propose a post-processing smoothing step that significantly improves the performance of some attribution methods.
  arXiv  Detail & Related papers  (2022-05-20T20:50:17Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.