LLMs for Legal Subsumption in German Employment Contracts

• URL: http://arxiv.org/abs/2507.01734v1
• Date: Wed, 02 Jul 2025 14:07:54 GMT
• Title: LLMs for Legal Subsumption in German Employment Contracts
• Authors: Oliver Wardas, Florian Matthes,
• Abstract summary: This study explores the use of Large Language Models and in-context learning to evaluate the legality of clauses in German employment contracts.<n>Our work evaluates the ability of different LLMs to classify clauses as "valid," "unfair," or "void" under three legal context variants.<n>Results show that full-text sources moderately improve performance, while examination guidelines significantly enhance recall for void clauses and weighted F1-Score, reaching 80%.
• Score: 3.3916160303055567
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Legal work, characterized by its text-heavy and resource-intensive nature, presents unique challenges and opportunities for NLP research. While data-driven approaches have advanced the field, their lack of interpretability and trustworthiness limits their applicability in dynamic legal environments. To address these issues, we collaborated with legal experts to extend an existing dataset and explored the use of Large Language Models (LLMs) and in-context learning to evaluate the legality of clauses in German employment contracts. Our work evaluates the ability of different LLMs to classify clauses as "valid," "unfair," or "void" under three legal context variants: no legal context, full-text sources of laws and court rulings, and distilled versions of these (referred to as examination guidelines). Results show that full-text sources moderately improve performance, while examination guidelines significantly enhance recall for void clauses and weighted F1-Score, reaching 80\%. Despite these advancements, LLMs' performance when using full-text sources remains substantially below that of human lawyers. We contribute an extended dataset, including examination guidelines, referenced legal sources, and corresponding annotations, alongside our code and all log files. Our findings highlight the potential of LLMs to assist lawyers in contract legality review while also underscoring the limitations of the methods presented.

Related papers

• Large Language Models in Legislative Content Analysis: A Dataset from the Polish Parliament [0.0]
  The research contributes to the advancement of NLP in the legal field, particularly in the Polish language.<n>It has been demonstrated that even commonly accessible data can be practically utilized for legislative content analysis.
  arXiv  Detail & Related papers  (2025-03-15T12:10:20Z)
• AI-assisted German Employment Contract Review: A Benchmark Dataset [3.3916160303055567]
  Recent advances in Natural Language Processing (NLP) hold promise for assisting in contract reviews.<n>Applying NLP techniques on legal text is particularly difficult due to the scarcity of expert-annotated datasets.<n>We release an anonymized and annotated benchmark dataset for legality and fairness review of German employment contract clauses.
  arXiv  Detail & Related papers  (2025-01-27T14:48:09Z)
• LegalAgentBench: Evaluating LLM Agents in Legal Domain [53.70993264644004]
  LegalAgentBench is a benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain.<n>LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge.
  arXiv  Detail & Related papers  (2024-12-23T04:02:46Z)
• Legal Evalutions and Challenges of Large Language Models [42.51294752406578]
  We use the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions. We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain.
  arXiv  Detail & Related papers  (2024-11-15T12:23:12Z)
• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval [16.29803062332164]
  We propose a few-shot approach where large language models assist in generating expert-aligned relevance judgments.<n>The proposed approach decomposes the judgment process into several stages, mimicking the workflow of human annotators.<n>It also ensures interpretable data labeling, providing transparency and clarity in the relevance assessment process.
  arXiv  Detail & Related papers  (2024-03-27T09:46:56Z)
• BLT: Can Large Language Models Handle Basic Legal Text? [44.89873147675516]
  GPT-4 and Claude perform poorly on basic legal text handling. Poor performance on benchmark casts into doubt their reliability as-is for legal practice. Fine-tuning on training set brings even a small model to near-perfect performance.
  arXiv  Detail & Related papers  (2023-11-16T09:09:22Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration [52.57055162778548]
  Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
  arXiv  Detail & Related papers  (2023-10-13T16:47:20Z)
• Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence [5.07013500385659]
  This paper explores Large Language Models' (LLMs) capabilities in applying tax law. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. Findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels.
  arXiv  Detail & Related papers  (2023-06-12T12:40:48Z)
• A Short Survey of Viewing Large Language Models in Legal Aspect [0.0]
  Large language models (LLMs) have transformed many fields, including natural language processing, computer vision, and reinforcement learning. The integration of LLMs into the legal field has also raised several legal problems, including privacy concerns, bias, and explainability.
  arXiv  Detail & Related papers  (2023-03-16T08:01:22Z)
• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
  We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding. We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
  arXiv  Detail & Related papers  (2021-05-09T09:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.