From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification

• URL: http://arxiv.org/abs/2310.11878v5
• Date: Fri, 16 Feb 2024 17:31:10 GMT
• Title: From Dissonance to Insights: Dissecting Disagreements in Rationale Construction for Case Outcome Classification
• Abstract summary: Case Outcome Classification (COC) must be accurate and trustworthy. Case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP.
• Score: 21.56347984160413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In legal NLP, Case Outcome Classification (COC) must not only be accurate but also trustworthy and explainable. Existing work in explainable COC has been limited to annotations by a single expert. However, it is well-known that lawyers may disagree in their assessment of case facts. We hence collect a novel dataset RAVE: Rationale Variation in ECHR1, which is obtained from two experts in the domain of international human rights law, for whom we observe weak agreement. We study their disagreements and build a two-level task-independent taxonomy, supplemented with COC-specific subcategories. To our knowledge, this is the first work in the legal NLP that focuses on human label variation. We quantitatively assess different taxonomy categories and find that disagreements mainly stem from underspecification of the legal context, which poses challenges given the typically limited granularity and noise in COC metadata. We further assess the explainablility of SOTA COC models on RAVE and observe limited agreement between models and experts. Overall, our case study reveals hitherto underappreciated complexities in creating benchmark datasets in legal NLP that revolve around identifying aspects of a case's facts supposedly relevant to its outcome.

Related papers

• Towards Explainable Adjudicative Variance: Quantifying Judicial Discretion via Gated Multi-Task Learning [9.159064044398315]
  Legal outcome prediction must disentangle objective case facts from adjudicative context.<n>We propose a Judge-Aware Gated Multi-Task Learning architecture that explicitly models this distinction.<n>We evaluate our approach on 13,937 UK Employment Tribunal decisions.
  arXiv  Detail & Related papers  (2026-06-25T14:14:27Z)
• GLIER: Generative Legal Inference and Evidence Ranking for Legal Case Retrieval [21.760917997629317]
  GLIER (Generative Legal Inference and Evidence Ranking) is a framework that reformulates retrieval as an inference process over latent legal variables.<n>First, a Joint Generative Inference module translates raw queries into latent legal indicators, including charges and legal elements.<n>Second, a Multi-View Evidence Fusion mechanism aggregates generative confidence with structural and lexical signals for precise ranking.
  arXiv  Detail & Related papers  (2026-04-26T16:02:59Z)
• Legal experts disagree with rationale extraction techniques for explaining ECtHR case outcome classification [9.334783986218232]
  Interpretability is critical for applications of large language models in the legal domain.<n>We propose a comparative analysis framework for model-agnostic interpretability techniques.<n>We show that the model's "reasons" for predicting a violation differ substantially from those of legal experts.
  arXiv  Detail & Related papers  (2026-01-18T14:03:17Z)
• Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics [49.3262123849242]
  We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset.<n>We convert court judgments into hierarchical trees of opposing parties' arguments and the court's conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces.
  arXiv  Detail & Related papers  (2025-11-30T18:32:43Z)
• ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation [56.79698529022327]
  Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution.<n>This paper explores the problem of legal claim generation based on the given case's facts.<n>We construct ClaimGen-CN, the first dataset for Chinese legal claim generation task.
  arXiv  Detail & Related papers  (2025-08-24T07:19:25Z)
• AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction [56.797874973414636]
  AnnoCaseLaw is a first-of-its-kind dataset of 471 meticulously annotated U.S. Appeals Court negligence cases. Our dataset lays the groundwork for more human-aligned, explainable Legal Judgment Prediction models. Results demonstrate that LJP remains a formidable task, with application of legal precedent proving particularly difficult.
  arXiv  Detail & Related papers  (2025-02-28T19:14:48Z)
• How Vital is the Jurisprudential Relevance: Law Article Intervened Legal Case Retrieval and Matching [31.378981566988063]
  Legal case retrieval (LCR) aims to automatically scour for comparable legal cases based on a given query. To address them, a daunting challenge is assessing the uniquely defined legal-rational similarity within the judicial domain. We propose an end-to-end model named LCM-LAI to solve the above challenges.
  arXiv  Detail & Related papers  (2025-02-25T15:29:07Z)
• LegalSeg: Unlocking the Structure of Indian Legal Judgments Through Rhetorical Role Classification [6.549338652948716]
  We introduce LegalSeg, the largest annotated dataset for this task, comprising over 7,000 documents and 1.4 million sentences, labeled with 7 rhetorical roles. Our results demonstrate that models incorporating broader context, structural relationships, and sequential sentence information outperform those relying solely on sentence-level features.
  arXiv  Detail & Related papers  (2025-02-09T10:07:05Z)
• ECtHR-PCR: A Dataset for Precedent Understanding and Prior Case Retrieval in the European Court of Human Rights [1.3723120574076126]
  We develop a prior case retrieval dataset based on judgements from the European Court of Human Rights (ECtHR) We benchmark different lexical and dense retrieval approaches with various negative sampling strategies. We find that difficulty-based negative sampling strategies were not effective for the PCR task.
  arXiv  Detail & Related papers  (2024-03-31T08:06:54Z)
• DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
  We introduce DELTA, a discriminative model designed for legal case retrieval. We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Our approach can outperform existing state-of-the-art methods in legal case retrieval.
  arXiv  Detail & Related papers  (2024-03-27T10:40:14Z)
• Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification [21.56347984160413]
  In legal decisions, split votes (SV) occur when judges cannot reach a unanimous decision. This paper explores split votes as naturally observable human disagreement and value pluralism. To our knowledge, this is the first systematic exploration of calibration to human judgements in legal NLP.
  arXiv  Detail & Related papers  (2024-02-11T14:21:37Z)
• Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement [3.537369004801589]
  We study the classification of legal reasoning according to jurisprudential philosophy. We use a novel dataset of historical United States Supreme Court opinions annotated by a team of domain experts. We find that generative models perform poorly when given instructions equal to the instructions presented to human annotators.
  arXiv  Detail & Related papers  (2023-10-27T19:27:59Z)
• MUSER: A Multi-View Similar Case Retrieval Dataset [65.36779942237357]
  Similar case retrieval (SCR) is a representative legal AI application that plays a pivotal role in promoting judicial fairness. Existing SCR datasets only focus on the fact description section when judging the similarity between cases. We present M, a similar case retrieval dataset based on multi-view similarity measurement and comprehensive legal element with sentence-level legal element annotations.
  arXiv  Detail & Related papers  (2023-10-24T08:17:11Z)
• Do Language Models Learn about Legal Entity Types during Pretraining? [4.604003661048267]
  We show that Llama2 performs well on certain entities and exhibits potential for substantial improvement with optimized prompt templates. Llama2 appears to frequently overlook syntactic cues, a shortcoming less present in BERT-based architectures.
  arXiv  Detail & Related papers  (2023-10-19T18:47:21Z)
• SemEval-2023 Task 11: Learning With Disagreements (LeWiDi) [75.85548747729466]
  We report on the second edition of the LeWiDi series of shared tasks. This second edition attracted a wide array of participants resulting in 13 shared task submission papers.
  arXiv  Detail & Related papers  (2023-04-28T12:20:35Z)
• SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval [75.05173891207214]
  Legal case retrieval plays a core role in the intelligent legal system. Most existing language models have difficulty understanding the long-distance dependencies between different structures. We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
  arXiv  Detail & Related papers  (2023-04-22T10:47:01Z)
• Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction [46.71918729837462]
  Given the fact description text of a legal case, legal judgment prediction aims to predict the case's charge, law article and penalty term. Previous studies fail to distinguish different classification errors with a standard cross-entropy classification loss. We propose a moco-based supervised contrastive learning to learn distinguishable representations. We further enhance the representation of the fact description with extracted crime amounts which are encoded by a pre-trained numeracy model.
  arXiv  Detail & Related papers  (2022-11-15T15:53:56Z)
• Equality before the Law: Legal Judgment Consistency Analysis for Fairness [55.91612739713396]
  In this paper, we propose an evaluation metric for judgment inconsistency, Legal Inconsistency Coefficient (LInCo) We simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups. We employ LInCo to explore the inconsistency in real cases and come to the following observations: (1) Both regional and gender inconsistency exist in the legal system, but gender inconsistency is much less than regional inconsistency.
  arXiv  Detail & Related papers  (2021-03-25T14:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.