Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases

• URL: http://arxiv.org/abs/2608.17168v1
• Date: Mon, 17 Aug 2026 22:11:22 GMT
• Title: Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
• Abstract summary: We investigate how LLMs reason in the context of legal case forecasting.<n>We use legal cases from the European Court of Human Rights as a testbed.<n>We find that the examined model scores far from ideal in legal reasoning.
• Score: 17.869207928898042
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning has become a standard technique and feature for contemporary LLMs; however, its application and quality in the context of demanding legal-oriented tasks, such as legal case forecasting, remain under explored. We investigate how LLMs reason in the context of legal case forecasting, using legal cases from the European Court of Human Rights (ECtHR) as a testbed. We evaluate OpenAI GPT 5.4, a recent top-tier LLM, by exploring alternative prompting strategies that are more or less suggestive of what counts as legally meaningful reasoning in the context of ECtHR jurisprudence. We present our findings derived from assessing the model's responses with both human and LLM evaluation. We find that the examined model scores far from ideal in legal reasoning, the model produces structurally complete but substantively shallow analyses, and that LLM-as-a-Judge evaluators are internally consistent yet align only weakly with our trained annotators, i.e., reliable but not a valid substitute for human evaluation. Overall, the expert-curated prompt leads to more comprehensive reasoning, which does not result in more accurate predictions compared to the other examined settings. Based on our findings, we urge the community not to rely solely on automated LLM-based evaluation and to avoid using task accuracy as an appropriate proxy for reasoning quality.

Related papers

• Toward Human Rights Benchmarking for LLMs: A Pilot Methodology [0.002301825353472044]
  HumRightsBench is the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of human rights law.<n>We find that model accuracy scores range considerably across legal reasoning tasks.
  arXiv  Detail & Related papers  (2026-08-10T22:09:05Z)
• LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation [9.599534367350227]
  This paper investigates the automatic generation and evaluation of legal propositions from decisions of the Court of Justice of the European Union using large language models (LLMs)<n>We introduce LP-Eval, a three-step evaluation rubric co-designed with legal experts that decomposes legal proposition quality into formal validity and substantive dimensions.<n>Our results show that LLMs can generate predominantly well-formed and high-quality propositions, while expert evaluations reveal higher quality for propositions derived from well established cases than from recent ones.
  arXiv  Detail & Related papers  (2026-05-19T13:10:39Z)
• Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training [75.98427023288052]
  Reasoning LLMs-as-Judges provide a promising path for extending the success of reasoning models to non-verifiable domains.<n>This study investigates the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment.<n>We find that reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs.
  arXiv  Detail & Related papers  (2026-03-12T17:57:06Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts [54.15982476754607]
  Large language models (LLMs) are now deployed at unprecedented scale, assisting millions of users in daily tasks.<n>This study defines complicit facilitation as the provision of guidance or support that enables illicit user instructions.<n>Using real-world legal cases and established legal frameworks, we construct an evaluation benchmark spanning 269 illicit scenarios and 50 illicit intents.
  arXiv  Detail & Related papers  (2025-11-25T16:01:31Z)
• LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appeal [34.008574054602356]
  The paper describes the structure of the exam, which includes a knowledge test on public procurement law and a written judgment.<n>Several LLMs were tested in closed-book and various Retrieval-Augmented Generation settings.<n>The results show that although the models achieved satisfactory scores in the knowledge test, none met the passing threshold in the practical written part.
  arXiv  Detail & Related papers  (2025-11-06T09:11:20Z)
• LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation [6.783926395409993]
  This paper introduces a novel, reference-free evaluation methodology that reflects how lawyers evaluate legal answers.<n>We show how our method correlates more closely with human expert evaluations and helps improve inter-annotator agreement.
  arXiv  Detail & Related papers  (2025-10-08T17:10:47Z)
• Quantitative LLM Judges [60.773734899532336]
  We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain.<n>The models are trained to improve the score of the original judge using its rationale and score.<n>Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling.
  arXiv  Detail & Related papers  (2025-06-03T14:44:23Z)
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization [69.23273504123941]
  We train judges to be robust to positional biases that arise in more complex evaluation settings.<n>We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work.<n>We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%.
  arXiv  Detail & Related papers  (2025-05-19T16:50:35Z)
• JudgeLRM: Large Reasoning Models as a Judge [80.07261839142548]
  We introduce JudgeLRM, a family of judgment-oriented Large Language Models (LLMs)<n>We find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios.<n>We show that JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models.
  arXiv  Detail & Related papers  (2025-03-31T02:18:51Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.