The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act

• URL: http://arxiv.org/abs/2606.18158v1
• Date: Tue, 16 Jun 2026 16:57:12 GMT
• Title: The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act
• Abstract summary: Large language models now produce legal text of at least median quality.<n>No existing benchmark can evaluate whether they perform doctrinal legal reasoning.<n>The EU AI Act makes "appropriate accuracy" a binding requirement for high-risk AI used in the judicial domain.
• Score: 0.08460698440162888
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models now produce legal text of at least median quality, yet no existing benchmark can evaluate whether they perform doctrinal legal reasoning, which forms the interpretive core of legal work, rather than the ancillary, paralegal tasks that most current legal-AI evaluations measure. This measurement gap is not only methodological but legal: the EU AI Act makes "appropriate accuracy" a binding requirement for high-risk AI used in the judicial domain, yet that requirement cannot acquire operational content without the very doctrinal-reasoning benchmark the field lacks.

Related papers

• Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice [0.0]
  Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice.<n>We argue that current benchmarks are not equipped to support this assumption because they evaluate legal reasoning over inputs that have already been preprocessed by legal experts.<n>We conclude by calling for legal benchmarks that directly measure robustness under pro se-like inputs so that access-to-justice claims about legal AI can become empirically testable.
  arXiv  Detail & Related papers  (2026-06-16T14:19:04Z)
• Bridging Legal Interpretation and Formal Logic: Faithfulness, Assumption, and the Future of AI Legal Reasoning [3.093890460224435]
  The growing adoption of large language models in legal practice brings both significant promise and serious risk.<n>This proposal presents a neuro-symbolic approach to legal AI that combines the expressive power of large language models with the rigor of formal verification.
  arXiv  Detail & Related papers  (2026-05-13T19:11:09Z)
• WisdomInterrogatory (LuWen): An Open-Source Legal Large Language Model Technical Report [55.27414605169639]
  We present WisdomInterrogatory (LuWen), an open-source Chinese legal language model built upon the Baichuan foundation model.<n>We evaluate LuWen on five representative legal tasks spanning both prediction and generation settings.
  arXiv  Detail & Related papers  (2026-04-08T06:59:07Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation [56.79698529022327]
  Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution.<n>This paper explores the problem of legal claim generation based on the given case's facts.<n>We construct ClaimGen-CN, the first dataset for Chinese legal claim generation task.
  arXiv  Detail & Related papers  (2025-08-24T07:19:25Z)
• GLARE: Agentic Reasoning for Legal Judgment Prediction [60.13483016810707]
  Legal judgment prediction (LJP) has become increasingly important in the legal field.<n>Existing large language models (LLMs) have significant problems of insufficient reasoning due to a lack of legal knowledge.<n>We introduce GLARE, an agentic legal reasoning framework that dynamically acquires key legal knowledge by invoking different modules.
  arXiv  Detail & Related papers  (2025-08-22T13:38:12Z)
• On Algorithmic Fairness and the EU Regulations [0.2538209532048867]
  The paper focuses on algorithmic fairness focusing on non-discrimination in the European Union (EU)<n>The paper demonstrates that correcting discriminatory biases in AI systems can be legally done under the EU regulations.<n>The paper contributes to the algorithmic fairness research with a few legal insights, enlarging and strengthening the growing research domain of compliance in AI engineering.
  arXiv  Detail & Related papers  (2024-11-13T06:23:54Z)
• The Fundamental Rights Impact Assessment (FRIA) in the AI Act: Roots, legal obligations and key elements for a model template [55.2480439325792]
  Article aims to fill existing gaps in the theoretical and methodological elaboration of the Fundamental Rights Impact Assessment (FRIA)<n>This article outlines the main building blocks of a model template for the FRIA.<n>It can serve as a blueprint for other national and international regulatory initiatives to ensure that AI is fully consistent with human rights.
  arXiv  Detail & Related papers  (2024-11-07T11:55:55Z)
• Authorized and Unauthorized Practices of Law: The Role of Autonomous Levels of AI Legal Reasoning [0.0]
  The legal field has sought to define Authorized Practices of Law (APL) versus Unauthorized Practices of Law (UPL) This paper explores a newly derived instrumental grid depicting the key characteristics underlying APL and UPL as they apply to the AILR autonomous levels.
  arXiv  Detail & Related papers  (2020-08-19T18:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.