LegalCiteBench: Evaluating Citation Reliability in Legal Language Models

• URL: http://arxiv.org/abs/2605.10186v1
• Date: Mon, 11 May 2026 08:37:54 GMT
• Title: LegalCiteBench: Evaluating Citation Reliability in Legal Language Models
• Abstract summary: LegalCiteBench is a benchmark for studying closed-book citation recovery, citation verification, and case matching in legal language models.<n>The benchmark covers five citation-centric tasks: citation retrieval, citation completion, citation error detection, case matching, and case verification and correction.
• Score: 14.281332347684872
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly integrated into legal drafting and research workflows, where incorrect citations or fabricated precedents can cause serious professional harm. Existing legal benchmarks largely emphasize statutory reasoning, contract understanding, or general legal question answering, but they do not directly study a central common-law failure mode: when asked to provide case authorities without external grounding, models may return plausible-looking but incorrect citations or cases. We introduce LegalCiteBench, a benchmark for studying closed-book citation recovery, citation verification, and case matching in legal language models. LegalCiteBench contains approximately 24K evaluation instances constructed from 1,000 real U.S. judicial opinions from the Case Law Access Project. The benchmark covers five citation-centric tasks: citation retrieval, citation completion, citation error detection, case matching, and case verification and correction. Across 21 LLMs, exact citation recovery remains highly challenging in this closed-book setting: even the strongest models score below 7/100 on citation retrieval and completion. Within the evaluated models, scale and legal-domain pretraining provide limited gains and do not resolve this difficulty. Models also frequently provide concrete but incorrect or low-overlap authorities under our evaluation protocol, with Misleading Answer Rates (MAR) exceeding 94% for 20 of 21 evaluated models on retrieval-heavy tasks. A prompt-only abstention experiment shows that explicit uncertainty instructions reduce some confident fabrication but do not improve citation correctness. LegalCiteBench is intended as a diagnostic framework for studying authority generation failures, verification behavior, and abstention when external grounding is absent, incomplete, or bypassed.

Related papers

• InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries [18.0624065854059]
  We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency.<n>We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes.<n>No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.
  arXiv  Detail & Related papers  (2026-08-20T16:14:47Z)
• Is this Citation on Point? [1.4303229099281964]
  We study proposition-level citation support verification through controlled perturbations of real legal citations.<n>Models catch 93%-100% of wrong-case corruptions.<n>They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs.
  arXiv  Detail & Related papers  (2026-08-12T20:28:55Z)
• Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice [0.0]
  Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice.<n>We argue that current benchmarks are not equipped to support this assumption because they evaluate legal reasoning over inputs that have already been preprocessed by legal experts.<n>We conclude by calling for legal benchmarks that directly measure robustness under pro se-like inputs so that access-to-justice claims about legal AI can become empirically testable.
  arXiv  Detail & Related papers  (2026-06-16T14:19:04Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation [75.27453112832093]
  We show that enforcing fine-grained citations degrades attribution quality by 16-276% compared to the best-performing granularity.<n>Our analysis suggests that fine-grained (sentence-level) citations disrupt necessary semantic dependencies for attributing evidence to answer claims.<n>Strikingly, citation-optimal granularity leads to substantial gains in attribution quality while preserving or even improving answer correctness.
  arXiv  Detail & Related papers  (2026-04-01T22:16:23Z)
• Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions [3.8449738927037207]
  We focus on implicit citation of the French Civil Code in first-instance court decisions.<n>We show that expert disagreement predicts model failures.<n>Despite these limits, reframing the task as top-k ranking and leveraging multi-model consensus yields 76% precision at k = 200 in an unsupervised setting.
  arXiv  Detail & Related papers  (2026-03-24T09:10:57Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• AR-BENCH: Benchmarking Legal Reasoning with Judgment Error Detection, Classification and Correction [30.1142279338937]
  Legal judgments may contain errors due to the complexity of case circumstances and the abstract nature of legal concepts.<n>Current legal AI research focuses on tasks like judgment prediction and legal document generation.<n>We introduce a novel task APPELLATE REVIEW, aiming to assess models' diagnostic reasoning and reliability in legal practice.
  arXiv  Detail & Related papers  (2026-01-30T09:22:32Z)
• CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval [5.305110876082343]
  CaseFacts is a benchmark for verifying legal claims against U.S. Supreme Court precedents.<n>The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled.
  arXiv  Detail & Related papers  (2026-01-23T23:41:46Z)
• Citation Failure: Definition, Analysis and Efficient Mitigation [56.09968229868067]
  Citations from LLM-based RAG systems are supposed to simplify response verification.<n>This does not hold for citation failure, when a model generates a helpful response, but fails to cite complete evidence.<n>We propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible.
  arXiv  Detail & Related papers  (2025-10-23T07:47:22Z)
• Conditioning Large Language Models on Legal Systems? Detecting Punishable Hate Speech [3.4300974012019148]
  This paper examines different approaches to conditioning Large Language Models (LLMs) at multiple levels of abstraction in legal systems to detect potentially punishable hate speech.<n>We focus on the task of classifying whether a specific social media posts falls under the criminal offense of incitement to hatred as prescribed by the German Criminal Code.<n>The results show that there is still a significant performance gap between models and legal experts in the legal assessment of hate speech, regardless of the level of abstraction with which the models were conditioned.
  arXiv  Detail & Related papers  (2025-06-03T15:50:27Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
  We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• CiteCaseLAW: Citation Worthiness Detection in Caselaw for Legal Assistive Writing [44.75251805925605]
  We introduce a labeled dataset of 178M sentences for citation-worthiness detection in the legal domain from the Caselaw Access Project (CAP) The performance of various deep learning models was examined on this novel dataset. The domain-specific pre-trained model tends to outperform other models, with an 88% F1-score for the citation-worthiness detection task.
  arXiv  Detail & Related papers  (2023-05-03T04:20:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.