What out-of-the-box LLMs can(t) do in law? A Turing test in Italian exams for lawyers, judges and notaries

• URL: http://arxiv.org/abs/2608.06166v1
• Date: Thu, 06 Aug 2026 15:33:21 GMT
• Title: What out-of-the-box LLMs can(t) do in law? A Turing test in Italian exams for lawyers, judges and notaries
• Abstract summary: The article reports on a blind Turing Test experiment, assessing the performance of out-of-the-box leading LLMs on three Italian legal professional exams.<n>The study identifies task-specific strengths, limitations and recurring legal failure patterns.
• Score: 0.8415089854734883
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The article reports on a blind Turing Test experiment, assessing the performance of out-of-the-box leading LLMs on three Italian legal professional exams: the Bar, Judges and Notary exams. Leading LLMs were asked to generate full written exam papers, which were made indistinguishable from human submissions and anonymously evaluated by expert examiners, using the same criteria applied in real examinations. Results reveal marked differences across both models and tasks. While some LLMs match or exceed top human performance in adversarial legal argumentation and doctrinal analysis, all models fail in the notary exam, which requires goal-directed legal planning under strict formal and substantive constraints. Beyond ranking models, the study identifies task-specific strengths, limitations and recurring legal failure patterns. Although limited to out-of-the-box systems, the findings provide qualitative evidence on the current scope and boundaries of the legal competence of LLMs across distinct professional tasks.

Related papers

• Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task [10.316445397110291]
  Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams.<n>This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction.
  arXiv  Detail & Related papers  (2026-04-26T14:15:43Z)
• Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training [75.98427023288052]
  Reasoning LLMs-as-Judges provide a promising path for extending the success of reasoning models to non-verifiable domains.<n>This study investigates the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment.<n>We find that reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs.
  arXiv  Detail & Related papers  (2026-03-12T17:57:06Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence [74.05988707492058]
  Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making.<n>Existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs)<n>We propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs.
  arXiv  Detail & Related papers  (2025-12-04T08:48:02Z)
• LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal [34.008574054602356]
  The paper describes the structure of the exam, which includes a knowledge test on public procurement law and a written judgment.<n>Several LLMs were tested in closed-book and various Retrieval-Augmented Generation settings.<n>The results show that although the models achieved satisfactory scores in the knowledge test, none met the passing threshold in the practical written part.
  arXiv  Detail & Related papers  (2025-11-06T09:11:20Z)
• Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoning [0.5308136763388956]
  We use India's public legal examinations as a transparent proxy.<n>Our benchmark assembles objective screens from top national and state exams.<n>We also include a lawyer-graded, paired-blinded study of long-form answers from the Supreme Court's Advocate-on-Record exam.
  arXiv  Detail & Related papers  (2025-10-19T10:04:29Z)
• Evaluating the Role of Large Language Models in Legal Practice in India [0.0]
  The integration of Artificial Intelligence into the legal profession raises significant questions about the capacity of Large Language Models to perform key legal tasks.<n>I empirically evaluate how well LLMs, such as GPT, Claude, and Llama, perform key legal tasks in the Indian context.<n>I conclude that while LLMs can augment certain legal tasks, human expertise remains essential for nuanced reasoning and the precise application of law.
  arXiv  Detail & Related papers  (2025-08-13T11:04:48Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
  We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences [76.73731245899454]
  We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience.<n>Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision.<n>This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the Intelligent Court''
  arXiv  Detail & Related papers  (2025-03-02T10:26:54Z)
• Legal Evalutions and Challenges of Large Language Models [42.51294752406578]
  We use the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions. We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain.
  arXiv  Detail & Related papers  (2024-11-15T12:23:12Z)
• BLT: Can Large Language Models Handle Basic Legal Text? [44.89873147675516]
  GPT-4 and Claude perform poorly on basic legal text handling. Poor performance on benchmark casts into doubt their reliability as-is for legal practice. Fine-tuning on training set brings even a small model to near-perfect performance.
  arXiv  Detail & Related papers  (2023-11-16T09:09:22Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
  We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding. We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
  arXiv  Detail & Related papers  (2021-05-09T09:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.