Identification of Regulatory Requirements Relevant to Business Processes: A Comparative Study on Generative AI, Embedding-based Ranking, Crowd and Expert-driven Methods

• URL: http://arxiv.org/abs/2401.02986v1
• Date: Tue, 2 Jan 2024 12:08:31 GMT
• Title: Identification of Regulatory Requirements Relevant to Business Processes: A Comparative Study on Generative AI, Embedding-based Ranking, Crowd and Expert-driven Methods
• Authors: Catherine Sai, Shazia Sadiq, Lei Han, Gianluca Demartini, Stefanie Rinderle-Ma
• Abstract summary: This work examines how legal and domain experts can be assisted in the assessment of relevant requirements. We compare an embedding-based NLP ranking method, a generative AI method using GPT-4, and a crowdsourced method with the purely manual method of creating labels by experts. A gold standard is created for both BPMN2.0 processes and matched to real-world requirements from multiple regulatory documents.
• Score: 10.899912290518648
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Organizations face the challenge of ensuring compliance with an increasing amount of requirements from various regulatory documents. Which requirements are relevant depends on aspects such as the geographic location of the organization, its domain, size, and business processes. Considering these contextual factors, as a first step, relevant documents (e.g., laws, regulations, directives, policies) are identified, followed by a more detailed analysis of which parts of the identified documents are relevant for which step of a given business process. Nowadays the identification of regulatory requirements relevant to business processes is mostly done manually by domain and legal experts, posing a tremendous effort on them, especially for a large number of regulatory documents which might frequently change. Hence, this work examines how legal and domain experts can be assisted in the assessment of relevant requirements. For this, we compare an embedding-based NLP ranking method, a generative AI method using GPT-4, and a crowdsourced method with the purely manual method of creating relevancy labels by experts. The proposed methods are evaluated based on two case studies: an Australian insurance case created with domain experts and a global banking use case, adapted from SAP Signavio's workflow example of an international guideline. A gold standard is created for both BPMN2.0 processes and matched to real-world textual requirements from multiple regulatory documents. The evaluation and discussion provide insights into strengths and weaknesses of each method regarding applicability, automation, transparency, and reproducibility and provide guidelines on which method combinations will maximize benefits for given characteristics such as process usage, impact, and dynamics of an application scenario.

Related papers

• Towards an Enforceable GDPR Specification [49.1574468325115]
  Privacy by Design (PbD) is prescribed by modern privacy regulations such as the EU's. One emerging technique to realize PbD is enforcement (RE) We present a set of requirements and an iterative methodology for creating formal specifications of legal provisions.
  arXiv  Detail & Related papers  (2024-02-27T09:38:51Z)
• Use case cards: a use case reporting framework inspired by the European AI Act [0.0]
  We propose a new framework for the documentation of use cases, that we call "use case cards" Unlike other documentation methodologies, we focus on the purpose and operational use of an AI system. The proposed framework is the result of a co-design process involving a relevant team of EU policy experts and scientists.
  arXiv  Detail & Related papers  (2023-06-23T15:47:19Z)
• Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs [65.9077733300329]
  Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents. LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.
  arXiv  Detail & Related papers  (2023-06-05T13:35:01Z)
• Flexible categorization for auditing using formal concept analysis and Dempster-Shafer theory [55.878249096379804]
  We study different ways to categorize according to different extents of interest in different financial accounts. The framework developed in this paper provides a formal ground to obtain and study explainable categorizations.
  arXiv  Detail & Related papers  (2022-10-31T13:49:16Z)
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems [76.79703106646967]
  We develop a value-based assessment framework that visualizes closeness and tensions between values. We give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
  arXiv  Detail & Related papers  (2022-05-09T19:28:32Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• Trustworthy Artificial Intelligence and Process Mining: Challenges and Opportunities [0.8602553195689513]
  We show that process mining can provide a useful framework for gaining fact-based visibility to AI compliance process execution. We provide for an automated approach to analyze, remediate and monitor uncertainty in AI regulatory compliance processes.
  arXiv  Detail & Related papers  (2021-10-06T12:50:47Z)
• Regulatory Compliance through Doc2Doc Information Retrieval: A case study in EU/UK legislation where text similarity has limitations [6.40476282000118]
  REG-IR is an application of document-to-document information retrieval. We show that fine-tuning a BERT model on an in-domain classification task produces the best representations for IR. We also show that neural re-rankers under-perform due to contradicting supervision, i.e., similar query-document pairs with opposite labels.
  arXiv  Detail & Related papers  (2021-01-26T11:38:15Z)
• Towards a Formal Framework for Partial Compliance of Business Processes [0.5156484100374059]
  In this paper, we formulate an evaluation framework to quantify the level of compliance of business processes across different levels of abstraction. Our approach can also add social value by making social services provided by local, state and federal governments more flexible and improving the lives of citizens.
  arXiv  Detail & Related papers  (2020-12-24T12:38:40Z)
• Extracting Procedural Knowledge from Technical Documents [1.0773368566852943]
  Procedures are an important knowledge component of documents that can be leveraged by cognitive assistants for automation, question-answering or driving a conversation. It is a challenging problem to parse big dense documents like product manuals, user guides to automatically understand which parts are talking about procedures and subsequently extract them.
  arXiv  Detail & Related papers  (2020-10-20T09:47:52Z)
• A Methodology for Creating AI FactSheets [67.65802440158753]
  This paper describes a methodology for creating the form of AI documentation we call FactSheets. Within each step of the methodology, we describe the issues to consider and the questions to explore. This methodology will accelerate the broader adoption of transparent AI documentation.
  arXiv  Detail & Related papers  (2020-06-24T15:08:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.