LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding

• URL: http://arxiv.org/abs/2508.07849v1
• Date: Mon, 11 Aug 2025 11:08:32 GMT
• Title: LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding
• Authors: Amrita Singh, H. Suhan Karaca, Aditya Joshi, Hye-young Paik, Jiaojiao Jiang,
• Abstract summary: Legal-specific LLMs consistently outperform general-purpose models.<n>CaseLaw-BERT and Contracts-BERT establish new SOTAs on two of the three tasks.
• Score: 3.281175376780409
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite advances in legal NLP, no comprehensive evaluation covering multiple legal-specific LLMs currently exists for contract classification tasks in contract understanding. To address this gap, we present an evaluation of 10 legal-specific LLMs on three English language contract understanding tasks and compare them with 7 general-purpose LLMs. The results show that legal-specific LLMs consistently outperform general-purpose models, especially on tasks requiring nuanced legal understanding. Legal-BERT and Contracts-BERT establish new SOTAs on two of the three tasks, despite having 69% fewer parameters than the best-performing general-purpose LLM. We also identify CaseLaw-BERT and LexLM as strong additional baselines for contract understanding. Our results provide a holistic evaluation of legal-specific LLMs and will facilitate the development of more accurate contract understanding systems.

Related papers

• LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence [74.05988707492058]
  Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making.<n>Existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs)<n>We propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs.
  arXiv  Detail & Related papers  (2025-12-04T08:48:02Z)
• Evaluating the Limits of Large Language Models in Multilingual Legal Reasoning [5.902003660308139]
  This work evaluates LLaMA and Gemini on multilingual legal and non-legal benchmarks.<n>We present an open-source, modular evaluation pipeline designed to support multilingual, task-diverse benchmarking.
  arXiv  Detail & Related papers  (2025-09-26T15:19:12Z)
• Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law [0.0]
  Large language models (LLMs) show promise, but their application in legal contexts demands higher accuracy, repeatability, and transparency.<n>We propose a neuro-symbolic approach that integrates LLMs' natural language understanding with logic-based reasoning.<n>As a legal document case study, we applied neuro-symbolic AI to coverage-related queries in insurance contracts.
  arXiv  Detail & Related papers  (2025-02-24T20:38:17Z)
• LegalAgentBench: Evaluating LLM Agents in Legal Domain [53.70993264644004]
  LegalAgentBench is a benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain.<n>LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge.
  arXiv  Detail & Related papers  (2024-12-23T04:02:46Z)
• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Benchmarking LLMs via Uncertainty Quantification [91.72588235407379]
  The proliferation of open-source Large Language Models (LLMs) has highlighted the urgent need for comprehensive evaluation methods. We introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs.
  arXiv  Detail & Related papers  (2024-01-23T14:29:17Z)
• BLT: Can Large Language Models Handle Basic Legal Text? [44.89873147675516]
  GPT-4 and Claude perform poorly on basic legal text handling. Poor performance on benchmark casts into doubt their reliability as-is for legal practice. Fine-tuning on training set brings even a small model to near-perfect performance.
  arXiv  Detail & Related papers  (2023-11-16T09:09:22Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• LAiW: A Chinese Legal Large Language Models Benchmark [17.66376880475554]
  General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. We are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice.
  arXiv  Detail & Related papers  (2023-10-09T11:19:55Z)
• LawBench: Benchmarking Legal Knowledge of Large Language Models [35.2812008533622]
  Large language models (LLMs) have demonstrated strong capabilities in various aspects. It is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels.
  arXiv  Detail & Related papers  (2023-09-28T09:35:59Z)
• Limits for Learning with Language Models [4.20859414811553]
  We show that large language models (LLMs) are unable to learn concepts beyond the first level of the Borel Hierarchy. LLMs will continue to operate without formal guarantees on tasks that require entailments and deep linguistic understanding.
  arXiv  Detail & Related papers  (2023-06-21T12:11:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.