Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios

• URL: http://arxiv.org/abs/2407.19760v2
• Date: Fri, 9 Aug 2024 10:43:18 GMT
• Title: Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios
• Abstract summary: GPT-4 consistently aligns more closely with progressive interpretations of the Constitution. Our experiments reveal a distinct tendency of GPT-4 to favor progressive legal interpretations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we conduct an empirical analysis of how large language models (LLMs), specifically GPT-4, interpret constitutional principles in complex decision-making scenarios. We examine rulings from the Italian Constitutional Court on bioethics issues that involve trade-offs between competing values and compare model-generated legal arguments on these issues to those presented by the State, the Court, and the applicants. Our results indicate that GPT-4 consistently aligns more closely with progressive interpretations of the Constitution, often overlooking competing values and mirroring the applicants' views rather than the more conservative perspectives of the State or the Court's moderate positions. Our experiments reveal a distinct tendency of GPT-4 to favor progressive legal interpretations, underscoring the influence of underlying data biases. We thus underscore the importance of testing alignment in real-world scenarios and considering the implications of deploying LLMs in decision-making processes.

Related papers

• Generating Legal Commentaries from Case Databases via Retrieval, Clustering, and Generation [8.81921009098836]
  We present a fully automated pipeline that transforms large collections of court decisions into legal commentaries for statutes.<n>We extract paragraph-level chunks, summarize their reasoning, and derive keywords, which are embedded and clustered.<n>Our results show that commentary-like argument mining from court decisions to generate reports can be refreshed within minutes at minimal cost.
  arXiv  Detail & Related papers  (2026-05-23T12:07:29Z)
• APPSI-139: A Parallel Corpus of English Application Privacy Policy Summarization and Interpretation [58.55537543016359]
  APPSI-139 is a high-quality English privacy policy corpus meticulously annotated by domain experts.<n>We propose TCSI-pp-V2, a hybrid privacy policy summarization and interpretation framework.
  arXiv  Detail & Related papers  (2026-04-30T07:58:24Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• Chinese Labor Law Large Language Model Benchmark [11.552694592413303]
  We present LabourLawLLM, a large language model tailored to Chinese labor law.<n>We also introduce LabourLawBench, a benchmark covering diverse labor-law tasks.<n> Experiments show that LabourLawLLM consistently outperforms general-purpose and existing legal-specific LLMs.
  arXiv  Detail & Related papers  (2026-01-15T01:27:29Z)
• Dissecting Judicial Reasoning in U.S. Copyright Damage Awards [0.21485350418225238]
  judicial reasoning in copyright damage awards poses a core challenge for computational legal analysis.<n>Federal courts follow the 1976 Copyright Act, their interpretations and factor weightings vary widely across jurisdictions.<n>This research introduces a novel discourse-based Large Language Model (LLM) methodology that integrates Rhetorical Structure Theory (RST) with an agentic workflow.
  arXiv  Detail & Related papers  (2026-01-14T13:09:16Z)
• Universal Legal Article Prediction via Tight Collaboration between Supervised Classification Model and LLM [42.11889345473452]
  Legal Article Prediction (LAP) is a critical task in legal text classification.<n>We propose Uni-LAP, a universal framework for legal article prediction.
  arXiv  Detail & Related papers  (2025-09-26T09:42:20Z)
• RLJP: Legal Judgment Prediction via First-Order Logic Rule-enhanced with Large Language Models [58.69183479148083]
  Legal Judgment Prediction (LJP) is a pivotal task in legal AI.<n>Existing LJP models integrate judicial precedents and legal knowledge for high performance.<n>But they neglect legal reasoning logic, a critical component of legal judgments requiring rigorous logical analysis.<n>This paper proposes a rule-enhanced legal judgment prediction framework based on first-order logic (FOL) formalism and comparative learning (CL)
  arXiv  Detail & Related papers  (2025-05-27T14:50:21Z)
• Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents [39.35255423087048]
  Legal rules encompass not only codified statutes but also implicit adjudicatory principles derived from precedents that contain discretionary norms, social morality, and policy.<n>We formalize Legal Rule Induction (LRI) as the task of deriving concise, generalizable doctrinal rules from sets of analogous precedents.<n>We introduce the first LRI benchmark, comprising 5,121 case sets (38,088 Chinese cases in total) for model tuning and 216 expert-annotated gold test sets.
  arXiv  Detail & Related papers  (2025-05-20T09:10:52Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [61.344330783528015]
  LEXam is a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• JudgeLRM: Large Reasoning Models as a Judge [65.14085339820795]
  We investigate whether Large Language Models (LLMs) judges truly benefit from enhanced reasoning capabilities. We introduce JudgeLRM, a family of judgment-oriented LLMs trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards.
  arXiv  Detail & Related papers  (2025-03-31T02:18:51Z)
• A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences [76.73731245899454]
  We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the Intelligent Court''
  arXiv  Detail & Related papers  (2025-03-02T10:26:54Z)
• AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction [56.797874973414636]
  AnnoCaseLaw is a first-of-its-kind dataset of 471 meticulously annotated U.S. Appeals Court negligence cases. Our dataset lays the groundwork for more human-aligned, explainable Legal Judgment Prediction models. Results demonstrate that LJP remains a formidable task, with application of legal precedent proving particularly difficult.
  arXiv  Detail & Related papers  (2025-02-28T19:14:48Z)
• How Vital is the Jurisprudential Relevance: Law Article Intervened Legal Case Retrieval and Matching [31.378981566988063]
  Legal case retrieval (LCR) aims to automatically scour for comparable legal cases based on a given query. To address them, a daunting challenge is assessing the uniquely defined legal-rational similarity within the judicial domain. We propose an end-to-end model named LCM-LAI to solve the above challenges.
  arXiv  Detail & Related papers  (2025-02-25T15:29:07Z)
• JUREX-4E: Juridical Expert-Annotated Four-Element Knowledge Base for Legal Reasoning [20.9057475907066]
  We present JUREX-4E, an expert-annotated four-element knowledge base covering 155 criminal charges.<n>We evaluate JUREX-4E on the Similar Charge Disambiguation task and apply it to Legal Case Retrieval.
  arXiv  Detail & Related papers  (2025-02-24T14:02:00Z)
• Legal Evalutions and Challenges of Large Language Models [42.51294752406578]
  We use the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions. We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain.
  arXiv  Detail & Related papers  (2024-11-15T12:23:12Z)
• Empirical analysis of binding precedent efficiency in Brazilian Supreme Court via case classification [2.6247415033465638]
  We empirically assess the legal impact of five binding precedents, 11, 14, 17, 26, and 37, at the highest Court level.<n>We identify five main hypotheses, which are found in different combinations in each of the precedents studied.
  arXiv  Detail & Related papers  (2024-07-09T16:17:16Z)
• Bridging Law and Data: Augmenting Reasoning via a Semi-Structured Dataset with IRAC methodology [22.740895683854568]
  This paper introduces LEGALSEMI, a benchmark specifically curated for legal scenario analysis. LEGALSEMI comprises 54 legal scenarios, each rigorously annotated by legal experts, based on the comprehensive IRAC (Issue, Rule, Application, Conclusion) framework. A series of experiments were conducted to assess the usefulness of LEGALSEMI for IRAC analysis.
  arXiv  Detail & Related papers  (2024-06-19T04:59:09Z)
• DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
  We introduce DELTA, a discriminative model designed for legal case retrieval. We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Our approach can outperform existing state-of-the-art methods in legal case retrieval.
  arXiv  Detail & Related papers  (2024-03-27T10:40:14Z)
• MUSER: A Multi-View Similar Case Retrieval Dataset [65.36779942237357]
  Similar case retrieval (SCR) is a representative legal AI application that plays a pivotal role in promoting judicial fairness. Existing SCR datasets only focus on the fact description section when judging the similarity between cases. We present M, a similar case retrieval dataset based on multi-view similarity measurement and comprehensive legal element with sentence-level legal element annotations.
  arXiv  Detail & Related papers  (2023-10-24T08:17:11Z)
• Explaining Legal Concepts with Augmented Large Language Models (GPT-4) [0.8924669503280334]
  We evaluate the performance of GPT-4 in generating factually accurate, clear and relevant explanations of terms in legislation. We found that the direct application of GPT-4 yields explanations that appear to be of very high quality on their surface. We found that the augmentation leads to improved quality, and appears to eliminate the issue of hallucination.
  arXiv  Detail & Related papers  (2023-06-15T21:58:18Z)
• Compatibility of Fairness Metrics with EU Non-Discrimination Laws: Demographic Parity & Conditional Demographic Disparity [3.5607241839298878]
  Empirical evidence suggests that algorithmic decisions driven by Machine Learning (ML) techniques threaten to discriminate against legally protected groups or create new sources of unfairness. This work aims at assessing up to what point we can assure legal fairness through fairness metrics and under fairness constraints. Our experiments and analysis suggest that AI-assisted decision-making can be fair from a legal perspective depending on the case at hand and the legal justification.
  arXiv  Detail & Related papers  (2023-06-14T09:38:05Z)
• Prototype-Based Interpretability for Legal Citation Prediction [16.660004925391842]
  We design the task with parallels to the thought-process of lawyers, i.e., with reference to both precedents and legislative provisions. After initial experimental results, we refine the target citation predictions with the feedback of legal experts. We introduce a prototype architecture to add interpretability, achieving strong performance while adhering to decision parameters used by lawyers.
  arXiv  Detail & Related papers  (2023-05-25T21:40:58Z)
• Perspectives on Large Language Models for Relevance Judgment [56.935731584323996]
  Large language models (LLMs) claim that they can assist with relevance judgments. It is not clear whether automated judgments can reliably be used in evaluations of retrieval systems.
  arXiv  Detail & Related papers  (2023-04-13T13:08:38Z)
• Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction [46.71918729837462]
  Given the fact description text of a legal case, legal judgment prediction aims to predict the case's charge, law article and penalty term. Previous studies fail to distinguish different classification errors with a standard cross-entropy classification loss. We propose a moco-based supervised contrastive learning to learn distinguishable representations. We further enhance the representation of the fact description with extracted crime amounts which are encoded by a pre-trained numeracy model.
  arXiv  Detail & Related papers  (2022-11-15T15:53:56Z)
• A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges [73.34944216896837]
  Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. We analyze 31 LJP datasets in 6 languages, present their construction process and define a classification method of LJP. We show the state-of-art results for 8 representative datasets from different court cases and discuss the open challenges.
  arXiv  Detail & Related papers  (2022-04-11T04:06:28Z)
• Equality before the Law: Legal Judgment Consistency Analysis for Fairness [55.91612739713396]
  In this paper, we propose an evaluation metric for judgment inconsistency, Legal Inconsistency Coefficient (LInCo) We simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups. We employ LInCo to explore the inconsistency in real cases and come to the following observations: (1) Both regional and gender inconsistency exist in the legal system, but gender inconsistency is much less than regional inconsistency.
  arXiv  Detail & Related papers  (2021-03-25T14:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.