Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation
- URL: http://arxiv.org/abs/2505.22041v1
- Date: Wed, 28 May 2025 07:03:46 GMT
- Title: Detecting Undesired Process Behavior by Means of Retrieval Augmented Generation
- Authors: Michael Grohs, Adrian Rebmann, Jana-Rebecca Rehse,
- Abstract summary: We propose an approach that requires neither a dedicated process model nor resource-intensive fine-tuning to detect undesired process behavior.<n>Instead, we use Retrieval Augmented Generation (RAG) to provide an LLM with direct access to a knowledge base.<n>Our evaluation shows that our approach outperforms fine-tuned LLMs in detecting undesired behavior.
- Score: 0.5461938536945723
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Conformance checking techniques detect undesired process behavior by comparing process executions that are recorded in event logs to desired behavior that is captured in a dedicated process model. If such models are not available, conformance checking techniques are not applicable, but organizations might still be interested in detecting undesired behavior in their processes. To enable this, existing approaches use Large Language Models (LLMs), assuming that they can learn to distinguish desired from undesired behavior through fine-tuning. However, fine-tuning is highly resource-intensive and the fine-tuned LLMs often do not generalize well. To address these limitations, we propose an approach that requires neither a dedicated process model nor resource-intensive fine-tuning to detect undesired process behavior. Instead, we use Retrieval Augmented Generation (RAG) to provide an LLM with direct access to a knowledge base that contains both desired and undesired process behavior from other processes, assuming that the LLM can transfer this knowledge to the process at hand. Our evaluation shows that our approach outperforms fine-tuned LLMs in detecting undesired behavior, demonstrating that RAG is a viable alternative to resource-intensive fine-tuning, particularly when enriched with relevant context from the event log, such as frequent traces and activities.
Related papers
- Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference [78.08901120841833]
We propose a method to detect the knowledge boundary of Visual Large Language Models (VLLMs)<n>We show that our method successfully depicts a VLLM's knowledge boundary based on which we are able to reduce indiscriminate retrieval while maintaining or improving the performance.
arXiv Detail & Related papers (2025-02-25T09:32:08Z) - Stream-Based Active Learning for Process Monitoring [1.2221087476416055]
This article presents a novel stream-based active learning strategy for Statistical process monitoring (SPM)
The ultimate goal is to optimize labeling resources constrained by a limited budget and dynamically update the possible OC states.
arXiv Detail & Related papers (2024-11-19T15:27:54Z) - Tractable Offline Learning of Regular Decision Processes [50.11277112628193]
This work studies offline Reinforcement Learning (RL) in a class of non-Markovian environments called Regular Decision Processes (RDPs)
Ins, the unknown dependency of future observations and rewards from the past interactions can be captured experimentally.
Many algorithms first reconstruct this unknown dependency using automata learning techniques.
arXiv Detail & Related papers (2024-09-04T14:26:58Z) - Mining Constraints from Reference Process Models for Detecting Best-Practice Violations in Event Log [1.389948527681755]
We propose a framework for mining declarative best-practice constraints from a reference model collection.
We demonstrate the capability of our framework to detect best-practice violations through an evaluation based on real-world process model collections and event logs.
arXiv Detail & Related papers (2024-07-02T15:05:37Z) - Adaptive Retention & Correction: Test-Time Training for Continual Learning [114.5656325514408]
A common problem in continual learning is the classification layer's bias towards the most recent task.<n>We name our approach Adaptive Retention & Correction (ARC)<n>ARC achieves an average performance increase of 2.7% and 2.6% on the CIFAR-100 and Imagenet-R datasets.
arXiv Detail & Related papers (2024-05-23T08:43:09Z) - Using Reinforcement Learning to Optimize Responses in Care Processes: A
Case Study on Aggression Incidents [0.8158530638728501]
We train a Markov decision process using event data from a care process.
The goal was to find optimal policies for staff members when clients are displaying any type of aggressive behavior.
Results showed that the policies derived from these algorithms are similar to the most frequent actions currently used.
arXiv Detail & Related papers (2023-10-02T08:43:29Z) - CAPE: Corrective Actions from Precondition Errors using Large Language
Models [8.547766794082184]
We propose a novel approach that attempts to propose corrective actions to resolve precondition errors during planning.
CAPE improves the quality of generated plans by leveraging few-shot reasoning from action preconditions.
Our improvements transfer to a Boston Dynamics Spot robot with a set of skills (specified in language) and associated preconditions, where CAPE improves the correctness metric of the executed task plans by 76.49% compared to SayCan.
arXiv Detail & Related papers (2022-11-17T23:14:51Z) - Detecting Surprising Situations in Event Data [0.45119235878273]
In existing work, it is usually assumed that the set of problematic process instances in which an undesirable outcome occurs is known prior or is easily detectable.
We formulate finding the process enhancement areas as a context-sensitive anomaly/outlier detection problem.
We aim to characterize those situations where process performance/outcome is significantly different from what was expected.
arXiv Detail & Related papers (2022-08-29T11:33:58Z) - A Monitoring and Discovery Approach for Declarative Processes Based on
Streams [0.0]
We present a discovery algorithm that extracts declarative processes as Dynamic Condition Response (DCR) graphs from event streams.
Streams are monitored to generate temporal representations of the process, later processed to generate declarative models.
arXiv Detail & Related papers (2022-08-10T14:25:35Z) - Chain of Thought Imitation with Procedure Cloning [129.62135987416164]
We propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations.
We show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations.
arXiv Detail & Related papers (2022-05-22T13:14:09Z) - DEALIO: Data-Efficient Adversarial Learning for Imitation from
Observation [57.358212277226315]
In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator.
Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms.
This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk.
We propose a more data-efficient IfO algorithm
arXiv Detail & Related papers (2021-03-31T23:46:32Z) - Certified Reinforcement Learning with Logic Guidance [78.2286146954051]
We propose a model-free RL algorithm that enables the use of Linear Temporal Logic (LTL) to formulate a goal for unknown continuous-state/action Markov Decision Processes (MDPs)
The algorithm is guaranteed to synthesise a control policy whose traces satisfy the specification with maximal probability.
arXiv Detail & Related papers (2019-02-02T20:09:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.