Is this Citation on Point?

• URL: http://arxiv.org/abs/2608.12571v1
• Date: Wed, 12 Aug 2026 20:28:55 GMT
• Title: Is this Citation on Point?
• Abstract summary: We study proposition-level citation support verification through controlled perturbations of real legal citations.<n>Models catch 93%-100% of wrong-case corruptions.<n>They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs.
• Score: 1.4303229099281964
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.

Related papers

• Who Checks the Citations? Benchmarking Legal Hallucination Detection [7.592193069681834]
  Attorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently fabricate citations.<n>This study evaluates whether AI-based systems can mitigate these errors by automatically detecting hallucinations.<n>We propose a taxonomy of legal citation hallucinations grounded in actual court filings and introduce a dataset of 1,300 brief excerpts containing injected errors.
  arXiv  Detail & Related papers  (2026-06-19T06:47:31Z)
• Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs [0.0]
  Empirical evaluation on 100 Ukrainian legal queries across five systems.<n>13-21% of citations hallucinated, with citation precision ranging from 0.791 to 0.873.<n>To reduce hallucinations without human annotation, we introduce Citation Grounding DPO.
  arXiv  Detail & Related papers  (2026-05-30T21:22:47Z)
• ImmigrationQA: A Source-Grounded Dataset and Small-Model Adaptation for U.S. Immigration Law [0.0]
  We describe the construction of ImmigrationQA, a source-grounded question-answering dataset of 17,058 pairs.<n>The corpus was assembled from 11 primary and secondary sources.<n>The fine-tuned model was evaluated against a held-out split of 993 pairs.
  arXiv  Detail & Related papers  (2026-05-28T21:36:18Z)
• LegalCiteBench: Evaluating Citation Reliability in Legal Language Models [14.281332347684872]
  LegalCiteBench is a benchmark for studying closed-book citation recovery, citation verification, and case matching in legal language models.<n>The benchmark covers five citation-centric tasks: citation retrieval, citation completion, citation error detection, case matching, and case verification and correction.
  arXiv  Detail & Related papers  (2026-05-11T08:37:54Z)
• Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation [75.27453112832093]
  We show that enforcing fine-grained citations degrades attribution quality by 16-276% compared to the best-performing granularity.<n>Our analysis suggests that fine-grained (sentence-level) citations disrupt necessary semantic dependencies for attributing evidence to answer claims.<n>Strikingly, citation-optimal granularity leads to substantial gains in attribution quality while preserving or even improving answer correctness.
  arXiv  Detail & Related papers  (2026-04-01T22:16:23Z)
• Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions [3.8449738927037207]
  We focus on implicit citation of the French Civil Code in first-instance court decisions.<n>We show that expert disagreement predicts model failures.<n>Despite these limits, reframing the task as top-k ranking and leveraging multi-model consensus yields 76% precision at k = 200 in an unsupervised setting.
  arXiv  Detail & Related papers  (2026-03-24T09:10:57Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Citation Failure: Definition, Analysis and Efficient Mitigation [56.09968229868067]
  Citations from LLM-based RAG systems are supposed to simplify response verification.<n>This does not hold for citation failure, when a model generates a helpful response, but fails to cite complete evidence.<n>We propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible.
  arXiv  Detail & Related papers  (2025-10-23T07:47:22Z)
• Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
  Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Missci is a novel argumentation theoretical model for fallacious reasoning. We present Missci as a dataset to test the critical reasoning abilities of large language models.
  arXiv  Detail & Related papers  (2024-06-05T12:11:10Z)
• Fact-Saboteurs: A Taxonomy of Evidence Manipulation Attacks against Fact-Verification Systems [80.3811072650087]
  We show that it is possible to subtly modify claim-salient snippets in the evidence and generate diverse and claim-aligned evidence. The attacks are also robust against post-hoc modifications of the claim. These attacks can have harmful implications on the inspectable and human-in-the-loop usage scenarios.
  arXiv  Detail & Related papers  (2022-09-07T13:39:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.