Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal

• URL: http://arxiv.org/abs/2607.04261v2
• Date: Fri, 10 Jul 2026 13:49:55 GMT
• Title: Shortcut Learning in Legal Judgment Prediction: Empirical Evidence from the UK Employment Tribunal
• Abstract summary: Current Legal Judgment Prediction (LJP) is constrained by its reliance on post-hoc judicial materials.<n>This paper investigates shortcut learning in this context by studying claim-level outcome prediction in UK Employment Tribunal (UKET) decisions.<n>We show that performance in LJP systems trained on post-hoc judicial text can be driven by the retrospective nature of the source material.
• Score: 2.4014236853596627
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current Legal Judgment Prediction (LJP) is constrained by its reliance on post-hoc judicial materials, increasing the likelihood that models perform retrospective classification rather than true forecasting. This paper empirically investigates shortcut learning in this context by studying claim-level outcome prediction in UK Employment Tribunal (UKET) decisions. Using a corpus of 33,158 individual claims, we predict outcomes from claim texts and LLM-extracted case summaries, evaluating models ranging from interpretable TF-IDF-based classifiers to black-box LLMs. While headline predictive performance figures appear strong, we demonstrate that such performance in LJP systems trained on post-hoc judicial text can be driven by the retrospective nature of the source material. Stratifying the test data by human judgments of leakage reveals that performance increases where outcome-revealing cues are embedded in the narrative. Moreover, a model trained on just the 4% of features identified as leakage achieves high performance, outperforming human experts. These findings substantiate concerns that LJP performance may be exaggerated by linguistic artefacts. Yet this vulnerability is not fatal to the research agenda. Instead, post-hoc judgments might be treated as potentially contaminated texts, requiring active auditing. Retraining models after masking leakage features results in only a negligible reduction in Macro-F1. Hence, while models will opportunistically exploit shortcuts when available, they remain capable of extracting useful predictive signals when these artefacts are removed.

Related papers

• Legibility is Not Interpretability: Comparing Judged and Actual Importance in Chain-Of-Thought Reasoning [48.96460246231164]
  Reasoning traces from chain-of-thought models appear to offer a legible window into how a model arrives at its answer.<n>We operationalize the importance of a reasoning step as its advantage.<n>We find that sufficiently capable LLMs can outperform a prevalence baseline but fall well short of a noise ceiling.
  arXiv  Detail & Related papers  (2026-09-03T17:59:08Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training [75.98427023288052]
  Reasoning LLMs-as-Judges provide a promising path for extending the success of reasoning models to non-verifiable domains.<n>This study investigates the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment.<n>We find that reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs.
  arXiv  Detail & Related papers  (2026-03-12T17:57:06Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Legal experts disagree with rationale extraction techniques for explaining ECtHR case outcome classification [9.334783986218232]
  Interpretability is critical for applications of large language models in the legal domain.<n>We propose a comparative analysis framework for model-agnostic interpretability techniques.<n>We show that the model's "reasons" for predicting a violation differ substantially from those of legal experts.
  arXiv  Detail & Related papers  (2026-01-18T14:03:17Z)
• Who's Your Judge? On the Detectability of LLM-Generated Judgments [37.318998323301365]
  Large Language Model (LLM)-based judgments leverage powerful LLMs to efficiently evaluate candidate content and provide judgment scores.<n>In this work, we propose and formalize the task of judgment detection.<n>We introduce textitJ-Detector, a lightweight and transparent neural detector augmented with explicitly extracted linguistic and LLM-enhanced features.
  arXiv  Detail & Related papers  (2025-09-29T17:54:57Z)
• Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models [3.552993426200889]
  This study investigates judgment prediction in a realistic scenario within the context of Indian judgments. We use a range of transformer-based models, including InLegalBERT, BERT, and XLNet, alongside LLMs such as Llama-2 and GPT-3.5 Turbo. Our experiments with LLMs reveal that GPT-3.5 Turbo excels in realistic scenarios, demonstrating robust performance in judgment prediction.
  arXiv  Detail & Related papers  (2024-10-14T14:22:12Z)
• The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal [0.41044181091229565]
  The study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Empirical results indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task.
  arXiv  Detail & Related papers  (2024-09-12T14:51:43Z)
• Understanding Privacy Risks of Embeddings Induced by Large Language Models [75.96257812857554]
  Large language models show early signs of artificial general intelligence but struggle with hallucinations. One promising solution is to store external knowledge as embeddings, aiding LLMs in retrieval-augmented generation. Recent studies experimentally showed that the original text can be partially reconstructed from text embeddings by pre-trained language models.
  arXiv  Detail & Related papers  (2024-04-25T13:10:48Z)
• Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset [2.7463268699570134]
  This study delves into the realm of explainability and fairness in Legal Judgement Prediction (LJP) models. We evaluate the explainability performance of state-of-the-art monolingual and multilingual BERT-based LJP models. We introduce a novel evaluation framework, Lower Court Insertion (LCI), which allows us to quantify the influence of lower court information on model predictions.
  arXiv  Detail & Related papers  (2024-02-26T20:42:40Z)
• Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration [52.57055162778548]
  Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
  arXiv  Detail & Related papers  (2023-10-13T16:47:20Z)
• Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction [46.71918729837462]
  Given the fact description text of a legal case, legal judgment prediction aims to predict the case's charge, law article and penalty term. Previous studies fail to distinguish different classification errors with a standard cross-entropy classification loss. We propose a moco-based supervised contrastive learning to learn distinguishable representations. We further enhance the representation of the fact description with extracted crime amounts which are encoded by a pre-trained numeracy model.
  arXiv  Detail & Related papers  (2022-11-15T15:53:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.