LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

• URL: http://arxiv.org/abs/2605.19815v1
• Date: Tue, 19 May 2026 13:10:39 GMT
• Title: LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation
• Abstract summary: This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs)<n>We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions.<n>Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones.
• Score: 9.599534367350227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs). We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions. Using this rubric, we release a dataset of two experts' annotations for 100 LLM-generated legal propositions. Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones. We further examine LLMs as evaluators and find that rubric-guided LLM judgments align more closely with expert assessments than direct overall scoring, but remain insensitive to finer-grained distinctions captured by human experts.

Related papers

• Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases [17.869207928898042]
  We investigate how LLMs reason in the context of legal case forecasting.<n>We use legal cases from the European Court of Human Rights as a testbed.<n>We find that the examined model scores far from ideal in legal reasoning.
  arXiv  Detail & Related papers  (2026-08-17T22:11:22Z)
• Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task [10.316445397110291]
  Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams.<n>This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction.
  arXiv  Detail & Related papers  (2026-04-26T14:15:43Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence [74.05988707492058]
  Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making.<n>Existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs)<n>We propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs.
  arXiv  Detail & Related papers  (2025-12-04T08:48:02Z)
• Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [49.3262123849242]
  We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset.<n>We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces.
  arXiv  Detail & Related papers  (2025-11-30T18:32:43Z)
• LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation [6.783926395409993]
  This paper introduces a novel, reference-free evaluation methodology that reflects how lawyers evaluate legal answers.<n>We show how our method correlates more closely with human expert evaluations and helps improve inter-annotator agreement.
  arXiv  Detail & Related papers  (2025-10-08T17:10:47Z)
• CLaw: Benchmarking Chinese Legal Knowledge in Large Language Models - A Fine-grained Corpus and Reasoning Analysis [13.067377421250557]
  Large Language Models (LLMs) are increasingly tasked with analyzing legal texts and citing relevant statutes.<n>This paper introduces CLaw, a novel benchmark specifically engineered to meticulously evaluate LLMs on Chinese legal knowledge and its application in reasoning.
  arXiv  Detail & Related papers  (2025-09-25T14:19:51Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Using Large Language Models for Legal Decision-Making in Austrian Value-Added Tax Law: An Experimental Study [0.0]
  This paper provides an experimental evaluation of the capability of large language models (LLMs) to assist in legal decision-making within the framework of Austrian and European Union value-added tax (VAT) law.
  arXiv  Detail & Related papers  (2025-07-11T10:19:56Z)
• LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation [68.26911563519162]
  Legal consultation is essential for safeguarding individual rights and ensuring access to justice.<n>Current systems fall short in handling the interactive and knowledge-intensive nature of real-world consultations.<n>LeCoDe is a real-world multi-turn benchmark dataset comprising 3,696 legal consultation dialogues with 110,008 dialogue turns.
  arXiv  Detail & Related papers  (2025-05-26T08:24:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.