Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice

• URL: http://arxiv.org/abs/2606.23716v1
• Date: Tue, 16 Jun 2026 14:19:04 GMT
• Title: Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice
• Abstract summary: Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice.<n>We argue that current benchmarks are not equipped to support this assumption because they evaluate legal reasoning over inputs that have already been preprocessed by legal experts.<n>We conclude by calling for legal benchmarks that directly measure robustness under pro se-like inputs so that access-to-justice claims about legal AI can become empirically testable.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice, including for people who cannot access lawyers in order to understand and exercise their legal rights. We argue that current benchmarks are not equipped to support this assumption because they evaluate legal reasoning over inputs that have already been preprocessed by legal experts, which measures the upper bound of model performance. Access to justice depends on a lower bound: how models perform when inputs come from pro se litigants, whose prompts may contain noisy narratives, buried facts, omissions, folk-legal assumptions, and surface-level errors. These degradations are comparable to conditions under which LLMs are known to degrade in the general machine learning literature, including long-context sensitivity, underspecification, hallucination, and typographical perturbations. We connect evidence from pro se literature with this body of machine learning research and present a small perturbation experiment on LEXam, a legal benchmark, to illustrate the gap between these two bounds. If model development continues to focus on benchmarks that measure only the upper bound, this gap may remain hidden or even widen. We conclude by calling for legal benchmarks that directly measure robustness under pro se-like inputs so that access-to-justice claims about legal AI can become empirically testable.

Related papers

• WisdomInterrogatory (LuWen): An Open-Source Legal Large Language Model Technical Report [55.27414605169639]
  We present WisdomInterrogatory (LuWen), an open-source Chinese legal language model built upon the Baichuan foundation model.<n>We evaluate LuWen on five representative legal tasks spanning both prediction and generation settings.
  arXiv  Detail & Related papers  (2026-04-08T06:59:07Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval [5.305110876082343]
  CaseFacts is a benchmark for verifying legal claims against U.S. Supreme Court precedents.<n>The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled.
  arXiv  Detail & Related papers  (2026-01-23T23:41:46Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [49.3262123849242]
  We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset.<n>We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces.
  arXiv  Detail & Related papers  (2025-11-30T18:32:43Z)
• Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities [15.35489310097019]
  CLAUSE is a first-of-its-kind benchmark designed to evaluate the fragility of an LLM's legal reasoning.<n>Our work outlines a path to identify and correct such reasoning failures in legal AI.
  arXiv  Detail & Related papers  (2025-11-01T00:51:21Z)
• Not ready for the bench: LLM legal interpretation is unstable and out of step with human judgments [2.8622281002418357]
  Recent scholarship has proposed that legal practitioners add large language models (LLMs) to their interpretive toolkit.<n>This work offers an empirical argument against LLM interpretation as recently practiced by legal scholars and federal judges.<n>Our investigation in English shows that models do not provide stable interpretive judgments.
  arXiv  Detail & Related papers  (2025-10-29T10:21:25Z)
• GLARE: Agentic Reasoning for Legal Judgment Prediction [60.13483016810707]
  Legal judgment prediction (LJP) has become increasingly important in the legal field.<n>Existing large language models (LLMs) have significant problems of insufficient reasoning due to a lack of legal knowledge.<n>We introduce GLARE, an agentic legal reasoning framework that dynamically acquires key legal knowledge by invoking different modules.
  arXiv  Detail & Related papers  (2025-08-22T13:38:12Z)
• LawLLM: Law Large Language Model for the US Legal System [43.13850456765944]
  We introduce the Law Large Language Model (LawLLM), a multi-task model specifically designed for the US legal domain. LawLLM excels at Similar Case Retrieval (SCR), Precedent Case Recommendation (PCR), and Legal Judgment Prediction (LJP) We propose customized data preprocessing techniques for each task that transform raw legal data into a trainable format.
  arXiv  Detail & Related papers  (2024-07-27T21:51:30Z)
• DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
  We introduce DELTA, a discriminative model designed for legal case retrieval. We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Our approach can outperform existing state-of-the-art methods in legal case retrieval.
  arXiv  Detail & Related papers  (2024-03-27T10:40:14Z)
• Prototype-Based Interpretability for Legal Citation Prediction [16.660004925391842]
  We design the task with parallels to the thought-process of lawyers, i.e., with reference to both precedents and legislative provisions. After initial experimental results, we refine the target citation predictions with the feedback of legal experts. We introduce a prototype architecture to add interpretability, achieving strong performance while adhering to decision parameters used by lawyers.
  arXiv  Detail & Related papers  (2023-05-25T21:40:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.