LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
- URL: http://arxiv.org/abs/2606.09389v1
- Date: Mon, 08 Jun 2026 12:04:47 GMT
- Title: LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
- Abstract summary: We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks.<n>It includes 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning.<n>It also includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework.
- Score: 42.88450102623113
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal question remains challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.
Related papers
- TriBench-Ko: Evaluating LLM Risks in Judicial Workflows [10.833772229369226]
TriBench-Ko is a Korean benchmark designed to evaluate potential deployment risks of large language models.<n>It covers four core tasks: summarization, precedent retrieval, legal issue extraction, and evidence analysis.<n>It jointly assesses model behavior across multiple deployment risk categories.
arXiv Detail & Related papers (2026-05-05T14:20:28Z) - AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction [30.1142279338937]
Legal judgments may contain errors due to the complexity of case circumstances and the abstract nature of legal concepts.<n>Current legal AI research focuses on tasks like judgment prediction and legal document generation.<n>We introduce a novel task APPELLATE REVIEW, aiming to assess models' diagnostic reasoning and reliability in legal practice.
arXiv Detail & Related papers (2026-01-30T09:22:32Z) - PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
arXiv Detail & Related papers (2026-01-23T11:36:10Z) - LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence [74.05988707492058]
Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making.<n>Existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs)<n>We propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs.
arXiv Detail & Related papers (2025-12-04T08:48:02Z) - LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
arXiv Detail & Related papers (2025-05-19T08:48:12Z) - Automatic Legal Writing Evaluation of LLMs [10.74636407144071]
oab-bench is a benchmark comprising 105 questions across seven areas of law from recent editions of the exam.<n>Claude-3.5 Sonnet achieves the best results with an average score of 7.93 out of 10, passing all 21 exams.<n>Our experiments show that frontier models like OpenAI's o1 achieve a strong correlation with human scores when evaluating approved exams.
arXiv Detail & Related papers (2025-04-29T22:16:39Z) - LegalAgentBench: Evaluating LLM Agents in Legal Domain [53.70993264644004]
LegalAgentBench is a benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain.<n>LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge.
arXiv Detail & Related papers (2024-12-23T04:02:46Z) - A Comprehensive Evaluation of Large Language Models on Legal Judgment
Prediction [60.70089334782383]
Large language models (LLMs) have demonstrated great potential for domain-specific applications.
Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks.
We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
arXiv Detail & Related papers (2023-10-18T07:38:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.