The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal

• URL: http://arxiv.org/abs/2409.08098v3
• Date: Sat, 26 Oct 2024 06:00:40 GMT
• Title: The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal
• Authors: Huiyuan Xie, Felix Steffek, Joana Ribeiro de Faria, Christine Carter, Jonathan Rutherford,
• Abstract summary: The study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Empirical results indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task.
• Score: 0.41044181091229565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper explores the intersection of technological innovation and access to justice by developing a benchmark for predicting case outcomes in the UK Employment Tribunal (UKET). To address the challenge of extensive manual annotation, the study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Comprehensive legal annotations cover facts, claims, precedent references, statutory references, case outcomes, reasons and jurisdiction codes. Facilitated by the CLC-UKET data, we examine a multi-class case outcome prediction task in the UKET. Human predictions are collected to establish a performance reference for model comparison. Empirical results from baseline models indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task. The performance of zero-shot LLMs can be enhanced by integrating task-related information into few-shot examples. We hope that the CLC-UKET dataset, along with human annotations and empirical findings, can serve as a valuable benchmark for employment-related dispute resolution.

Related papers

• Low-Resource Court Judgment Summarization for Common Law Systems [32.13166048504629]
  We present CLSum, the first dataset for summarizing multi-jurisdictional common law court judgment documents. This is the first court judgment summarization work adopting large language models (LLMs) in data augmentation, summary generation, and evaluation.
  arXiv  Detail & Related papers  (2024-03-07T12:47:42Z)
• Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement [3.537369004801589]
  We study the classification of legal reasoning according to jurisprudential philosophy. We use a novel dataset of historical United States Supreme Court opinions annotated by a team of domain experts. We find that generative models perform poorly when given instructions equal to the instructions presented to human annotators.
  arXiv  Detail & Related papers  (2023-10-27T19:27:59Z)
• Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks [65.21536453075275]
  We focus on the summarization task and investigate the membership inference (MI) attack. We exploit text similarity and the model's resistance to document modifications as potential MI signals. We discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.
  arXiv  Detail & Related papers  (2023-10-20T05:44:39Z)
• Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration [52.57055162778548]
  Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
  arXiv  Detail & Related papers  (2023-10-13T16:47:20Z)
• A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents [0.5812284760539713]
  We define this problem as "scarce annotated legal documents" We propose a deep-learning-based classification framework which we call MESc. We also propose an explanation extraction algorithm named ORSE.
  arXiv  Detail & Related papers  (2023-09-19T12:18:28Z)
• Evaluating and Explaining Large Language Models for Code Using Syntactic Structures [74.93762031957883]
  This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes. We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
  arXiv  Detail & Related papers  (2023-08-07T18:50:57Z)
• LMGQS: A Large-scale Dataset for Query-focused Summarization [77.6179359525065]
  We convert four generic summarization benchmarks into a new QFS benchmark dataset, LMGQS. We establish baselines with state-of-the-art summarization models. We achieve state-of-the-art zero-shot and supervised performance on multiple existing QFS benchmarks.
  arXiv  Detail & Related papers  (2023-05-22T14:53:45Z)
• CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding [15.685369142294693]
  CaseEncoder is a legal document encoder that leverages fine-grained legal knowledge in both the data sampling and pre-training phases. CaseEncoder significantly outperforms both existing general pre-training models and legal-specific pre-training models in zero-shot legal case retrieval.
  arXiv  Detail & Related papers  (2023-05-09T12:40:19Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)
• Understanding Translationese in Cross-Lingual Summarization [106.69566000567598]
  Cross-lingual summarization (MS) aims at generating a concise summary in a different target language. To collect large-scale CLS data, existing datasets typically involve translation in their creation. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese.
  arXiv  Detail & Related papers  (2022-12-14T13:41:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.