From Legal Text to Executable Decision Models: Evaluating Structured Representations for Legal Decision Model Generation

• URL: http://arxiv.org/abs/2604.17153v1
• Date: Sat, 18 Apr 2026 21:36:48 GMT
• Title: From Legal Text to Executable Decision Models: Evaluating Structured Representations for Legal Decision Model Generation
• Abstract summary: We use a unique real-world dataset that pairs production-grade decision models with legal text from the Dutch Environment and Planning Act.<n>These models power the Omgevingsloket government platform, where citizens check permit requirements for environmental activities.<n>We study whether intermediate structured representations can improve LLM-based generation of executable decision models from legal text.
• Score: 0.3519650551107762
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Transforming legal text into executable decision logic is a longstanding challenge in legal informatics. With the rise of LLMs, this task has gained renewed interest, but remains challenging due to requiring extensive manual coding and evaluation. We use a unique real-world dataset that pairs production-grade decision models with legal text from the Dutch Environment and Planning Act. These models power the Omgevingsloket government platform, where citizens check permit requirements for environmental activities. We study whether intermediate structured representations can improve LLM-based generation of executable decision models from legal text. We compare four input conditions: raw legal text, text enriched with semantic role labels, text enriched with input and output constraints, and text enriched with both. We evaluate along two dimensions: structural evaluation, through similarity to gold decision models with graph kernels and graphs' descriptive statistics, and outcome evaluation, through functional equivalence by executing models on pre-configured test scenarios. Our findings show that I/O constraints provide the dominant improvement (+37-54% similarity over baseline), while semantic role labels show modest improvements. Outcome evaluation shows that generated models match the gold standard on 51-53% of test scenarios, even though generated models are typically smaller and simpler. We find LLMs eliminate redundant pass-through logic that comprises up to 45-55% of nodes. Importantly, structural similarity and outcome equivalence are complementary: structural similarity does not guarantee outcome equivalence, and vice versa. To facilitate reproducibility, we publicly release our dataset of 95 production decision models with associated legal text and all experimental code.

Related papers

• ANNOTARES: A Dataset for Extracting Logical Structures from German Statutory Texts [4.897636783430679]
  We introduce the task of identifying and segmenting legal conditions (Tatbestand) and legal consequences (Rechtsfolge) within German statutory texts.<n>To support this task, we present ANNOTARES, a novel dataset comprising German law texts with span-level annotations.
  arXiv  Detail & Related papers  (2026-08-04T16:28:53Z)
• DECODEM: Data Extraction from Corporate Organizational Documents via Enhanced Methods [2.0789144738685956]
  This paper introduces DECODEM, a set of benchmark datasets for evaluating the automated extraction of corporate governance variables from organizational documents.<n>The benchmarks pair randomly sampled corporate charters and bylaws with high-quality human annotations covering a range of governance provisions commonly studied in empirical work.<n>The results show that automated extraction is feasible at a high level of accuracy for many provisions, with median performance near the upper bound across approaches.
  arXiv  Detail & Related papers  (2026-07-17T11:48:31Z)
• ViLegalNLI: Natural Language Inference for Vietnamese Legal Texts [5.44919589318124]
  We introduce ViLegalNLI, the first large-scale Vietnamese Natural Language Inference dataset specifically constructed for the legal domain.<n>The dataset consists of 42,012 premise-hypothesis pairs derived from official statutory documents and annotated with binary inference labels.<n>We conduct extensive experiments on the ViLegalNLI using multilingual models, Vietnamese-specific pretrained language models, and instruction-tuned large language models.
  arXiv  Detail & Related papers  (2026-04-30T18:16:14Z)
• LLMStructBench: Benchmarking Large Language Model Structured Data Extraction [1.338174941551702]
  We present a novel benchmark for evaluating Large Language Models (LLMs)<n>Our open dataset comprises diverse, manually verified parsing scenarios of varying complexity.<n>We show that choosing the right prompting strategy is more important than standard attributes such as model size.
  arXiv  Detail & Related papers  (2026-02-16T13:37:58Z)
• Modeling the Diachronic Evolution of Legal Norms: An LRMoo-Based, Component-Level, Event-Centric Approach to Legal Knowledge Graphs [0.0]
  This paper proposes a structured, temporal modeling pattern grounded in the LRMoo to address this need.<n>We show how this event-generative architecture enables the precise, deterministic retrieval and reconstruction of any part of a legal text as it existed on a specific date.
  arXiv  Detail & Related papers  (2025-06-09T15:18:36Z)
• An Ontology-Driven Graph RAG for Legal Norms: A Structural, Temporal, and Deterministic Approach [0.0]
  RAG systems in the legal domain face a critical challenge: standard, flat-text retrieval is blind to the hierarchical, diachronic, and causal structure of law, leading to anachronistic and unreliable answers.<n>This paper introduces the Structure-Aware Temporal Graph RAG (SAT-Graph RAG), an ontology-driven framework designed to overcome these limitations by explicitly modeling the formal structure and diachronic nature of legal norms.
  arXiv  Detail & Related papers  (2025-04-29T18:36:57Z)
• Attribution analysis of legal language as used by LLM [0.0]
  We use two publicly-available legal datasets, a simpler binary classification task of overruling'' texts, and a more elaborate multiple choice task identifying holding'' judicial decisions.<n>We find that while all models correctly classify some test examples from the casehold task, other examples can only be identified by only one, model, and attribution can be used to highlight the reasons for this.
  arXiv  Detail & Related papers  (2025-01-28T22:48:29Z)
• Analysing Zero-Shot Readability-Controlled Sentence Simplification [54.09069745799918]
  We investigate how different types of contextual information affect a model's ability to generate sentences with the desired readability.<n>Results show that all tested models struggle to simplify sentences due to models' limitations and characteristics of the source sentences.<n>Our experiments also highlight the need for better automatic evaluation metrics tailored to RCTS.
  arXiv  Detail & Related papers  (2024-09-30T12:36:25Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)
• MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generation [102.20036684996248]
  We propose MURMUR, a neuro-symbolic modular approach to text generation from semi-structured data with multi-step reasoning. We conduct experiments on two data-to-text generation tasks like WebNLG and LogicNLG.
  arXiv  Detail & Related papers  (2022-12-16T17:36:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.