From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry

• URL: http://arxiv.org/abs/2608.21317v1
• Date: Fri, 21 Aug 2026 17:27:13 GMT
• Title: From Regulation to Implementation: A Critical Evaluation of LLM-Assisted Regulatory Compliance in Industry
• Abstract summary: This study investigates how data extraction instructions and vagueness impact the quality and consistency of compliance artifacts.<n>The resulting artifacts are evaluated by benchmarking different models against manually created ground-to-ground schemas.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The European Union (EU) has emerged as a leading regulatory body in the development of sustainability and privacy regulations. While new regulation requirements vary, many include a documentation artifact to ensure compliance. Notably, the Ecodesign for Sustainable Products Regulation (ESPR) introduces Digital Product Passports (DPPs) for life cycle transparency, while the General Data Protection Regulation (GDPR) mandates Data Protection Impact Assessments (DPIAs) to mitigate privacy risks. Creating these compliance artifacts, however, is challenging. Industrial data, which often exists in heterogeneous formats and is scattered across company and supplier systems, is required for DPPs and can be difficult to extract into compliant DPP formatting. Furthermore, DPIA documents require interdisciplinary expertise and follow no standardized format, making development difficult for novel systems. To address the particular complexity of compliance artifact creation for both regulations, researchers have proposed the use of LLMs in the generation process; however, the impact of the aforementioned problems on the output of these systems is largely unaddressed. This work investigates the existing research gap by exploring how data extraction instructions and regulatory vagueness impact the quality and consistency of LLM-produced compliance artifacts. The resulting artifacts are evaluated by benchmarking different models against manually created ground-truth schemas. The results reveal that less strict guidelines, such as DPIA formatting, require higher context prompts to maintain consistency and completeness. Stricter guidelines, such as formatting for Digital Battery Passports (DBP), result in consistent results regardless of prompt context, but may lead to more hallucinations in the output

Related papers

• Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance [47.37396434232136]
  This paper introduces DGEval, the first benchmark for evaluating knowledge of IMDG Amendment 42-24.<n>It comprises 1,678 questions across multiple-choice, open-ended, DGL lookup, and regulatory identification tasks.<n>We evaluate 13 models from six providers across multiple thinking configurations.
  arXiv  Detail & Related papers  (2026-08-21T12:32:31Z)
• CTRAG: An In-Context Retrieval-based Framework for Automated Compliance Checking using LLMs [3.4983716329137344]
  We present CTRAG, a novel Retrieval-Augmented Generation (RAG) pipeline designed for automated compliance checking.<n>CTRAG achieves highly accurate, document-informed compliance verification, even in cases of indirect compliance through third-party services.<n> Empirical evaluations demonstrate significant improvements, with CTRAG achieving an F1-score of 78% and a recall of 85% in the final deployed configuration.
  arXiv  Detail & Related papers  (2026-08-03T16:40:16Z)
• ReCon: A Resource-Constrained Benchmark for LLM-Based Cybersecurity Compliance Across Ingestion and Retrieval Pipelines [0.6773121102591491]
  Governance failure is one of the major reasons for a weakened cybersecurity posture.<n>The policy document must be in compliance with international or national standards.<n>It is often observed that policy documents are often incomplete with respect to industry standards or regulations.
  arXiv  Detail & Related papers  (2026-07-24T19:53:00Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• Judging by the Rules: Compliance-Aligned Framework for Modern Slavery Statement Monitoring [24.13989765643719]
  Modern slavery affects millions of people worldwide, and regulatory frameworks such as Modern Slavery Acts now require companies to publish detailed disclosures.<n>These statements are often vague and inconsistent, making manual review time-consuming and difficult to scale.<n>We propose a novel framework that harnesses AI for rule-level compliance verification while preserving expert oversight.
  arXiv  Detail & Related papers  (2025-11-11T03:41:44Z)
• Analyzing and Internalizing Complex Policy Documents for LLM Agents [53.14898416858099]
  Large Language Model (LLM)-based agentic systems rely on in-context policy documents encoding diverse business rules.<n>This motivates developing internalization methods that embed policy documents into model priors while preserving performance.<n>We introduce CC-Gen, an agentic benchmark generator with Controllable Complexity across four levels.
  arXiv  Detail & Related papers  (2025-10-13T16:30:07Z)
• An Automated LLM-based Pipeline for Asset-Level Database Creation to Assess Deforestation Impact [0.0]
  The European Union Deforestation Regulation (EUDR) requires companies to prove their products do not contribute to deforestation.<n>Current databases lack the necessary detail, relying heavily on broad financial metrics and manual data collection.<n>This study presents an automated, end-to-end data extraction pipeline that uses LLMs to create, clean, and validate structured databases.
  arXiv  Detail & Related papers  (2025-05-05T15:17:27Z)
• RIRAG: Regulatory Information Retrieval and Answer Generation [51.998738311700095]
  We introduce a task of generating question-passages pairs, where questions are automatically created and paired with relevant regulatory passages.<n>We create the ObliQA dataset, containing 27,869 questions derived from the collection of Abu Dhabi Global Markets (ADGM) financial regulation documents.<n>We design a baseline Regulatory Information Retrieval and Answer Generation (RIRAG) system and evaluate it with RePASs, a novel evaluation metric.
  arXiv  Detail & Related papers  (2024-09-09T14:44:19Z)
• Towards an Enforceable GDPR Specification [49.1574468325115]
  Privacy by Design (PbD) is prescribed by modern privacy regulations such as the EU's. One emerging technique to realize PbD is enforcement (RE) We present a set of requirements and an iterative methodology for creating formal specifications of legal provisions.
  arXiv  Detail & Related papers  (2024-02-27T09:38:51Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.