Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs

• URL: http://arxiv.org/abs/2606.00898v1
• Date: Sat, 30 May 2026 21:22:47 GMT
• Title: Citation Grounding: Detecting and Reducing LLM Citation Hallucinations via Legal Citation Graphs
• Abstract summary: Empirical evaluation on 100 Ukrainian legal queries across five systems.<n>13-21% of citations hallucinated, with citation precision ranging from 0.791 to 0.873.<n>To reduce hallucinations without human annotation, we introduce Citation Grounding DPO.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models systematically hallucinate legal citations -- fabricating statute references, citing repealed provisions, and confusing jurisdictions -- yet no automated method exists to measure or reduce this behavior at scale. We propose citation grounding (CG), a metric that verifies LLM-generated legal citations against a ground-truth citation graph extracted from 100.8 million Ukrainian court decisions (502 million edges, 21,736 unique statute nodes). CG decomposes into three components -- citation precision (does the cited provision exist?), citation relevance (is it contextually appropriate?), and citation temporality (was it valid at the relevant date?) -- enabling differential diagnosis of hallucination types. Empirical evaluation on 100 Ukrainian legal queries across five systems -- four commercial LLMs via AWS Bedrock (Claude Haiku 4.5, Mistral Pixtral Large, Amazon Nova Pro/Lite) and one RAG-augmented production system -- reveals CG ranging from 0.791 to 0.873, with 13-21% of citations hallucinated. To reduce hallucinations without human annotation, we introduce Citation Grounding DPO (CG-DPO): a method that constructs preference pairs algorithmically by corrupting verified citations from real court decisions via four targeted strategies. On a dataset of 2,244 court decisions, a Qwen2.5-7B-Instruct model fine-tuned with LoRA achieves 98.5% mean validation accuracy in distinguishing correct from corrupted citations (rewards margin +14.9, std < 0.3 pp across 3 seeds). The citation graph, evaluation framework, and CG-DPO dataset are released as open resources.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Who is the Agent to Blame? Localizing Faithfulness and Citation Mistakes in Agentic Deep Research [75.8770212601511]
  Deep research (DR) systems produce long-form cited reports by orchestrating multiple agents that search and synthesize information from the web.<n> Citations are the primary mechanism for evaluating the faithfulness of these reports, yet current DR systems exhibit poor citation recall.<n>We propose an evaluation method that pinpoints which agent introduced each error by locally testing agent invocations for faithfulness and verifiability relative to their own inputs.
  arXiv  Detail & Related papers  (2026-08-25T09:34:18Z)
• Is this Citation on Point? [1.4303229099281964]
  We study proposition-level citation support verification through controlled perturbations of real legal citations.<n>Models catch 93%-100% of wrong-case corruptions.<n>They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs.
  arXiv  Detail & Related papers  (2026-08-12T20:28:55Z)
• Who Checks the Citations? Benchmarking Legal Hallucination Detection [7.592193069681834]
  Attorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently fabricate citations.<n>This study evaluates whether AI-based systems can mitigate these errors by automatically detecting hallucinations.<n>We propose a taxonomy of legal citation hallucinations grounded in actual court filings and introduce a dataset of 1,300 brief excerpts containing injected errors.
  arXiv  Detail & Related papers  (2026-06-19T06:47:31Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• Automatic Construction of a Legal Citation Graph from 100 Million Ukrainian Court Decisions: Large-Scale Extraction, Topological Analysis, and Ontology-Driven Clustering [0.0]
  Citation edges extracted from 100.7 million Ukrainian court decisions reveal judicial citation structure encodes legal domain boundaries without supervision.<n>We construct the first large-scale citation graph from the completeSR registry (99.5 million full texts, 1.1 TB)
  arXiv  Detail & Related papers  (2026-05-14T19:42:20Z)
• LegalCiteBench: Evaluating Citation Reliability in Legal Language Models [14.281332347684872]
  LegalCiteBench is a benchmark for studying closed-book citation recovery, citation verification, and case matching in legal language models.<n>The benchmark covers five citation-centric tasks: citation retrieval, citation completion, citation error detection, case matching, and case verification and correction.
  arXiv  Detail & Related papers  (2026-05-11T08:37:54Z)
• Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection [30.352741291341843]
  Existing detectors reduce to binary found/not-found decisions, offering little field-level signal to auditors.<n>We introduce a 12-code taxonomy spanning Real, Potential, and Hallucinated citations.<n>CiteTracer extracts citations from PDF and Bib, fetches evidence through lookup URL, scholar connectors, and web search.<n>It reaches 97.1% accuracy on the synthetic benchmark, with class-level F1 scores of 97.0, 95.8, and 98.5 for Real, Potential, and Hallucinated, respectively, and fabricates hallucinations on the real-world set without abstaining.
  arXiv  Detail & Related papers  (2026-05-09T00:53:24Z)
• Are Finer Citations Always Better? Rethinking Granularity for Attributed Generation [75.27453112832093]
  We show that enforcing fine-grained citations degrades attribution quality by 16-276% compared to the best-performing granularity.<n>Our analysis suggests that fine-grained (sentence-level) citations disrupt necessary semantic dependencies for attributing evidence to answer claims.<n>Strikingly, citation-optimal granularity leads to substantial gains in attribution quality while preserving or even improving answer correctness.
  arXiv  Detail & Related papers  (2026-04-01T22:16:23Z)
• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• How LLMs Cite and Why It Matters: A Cross-Model Audit of Reference Fabrication in AI-Assisted Academic Writing and Methods to Detect Phantom Citations [1.0829694003408499]
  Large language models (LLMs) have been noted to fabricate scholarly citations, yet the scope of this behavior remains poorly quantified.<n>We present one of the largest hallucination citation audits to date, in which 10 commercially deployed LLMs were prompted across four academic domains.<n>Our results show that the observed hallucination rates span a fivefold range (between 11.4% and 56.8%) and are strongly shaped by model, domain, and prompt framing.
  arXiv  Detail & Related papers  (2026-02-07T00:14:00Z)
• Aligning Large Language Model Behavior with Human Citation Preferences [45.80355133880463]
  We study the relationship between human citation preferences and large-scale language models (LLMs) behavior.<n>Our results show that humans most frequently seek citations for medical text, and stronger models display a similar tendency.
  arXiv  Detail & Related papers  (2026-02-05T02:02:43Z)
• FactNet: A Billion-Scale Knowledge Graph for Multilingual Factual Grounding [81.2130536158575]
  LLMs exhibit remarkable fluency, their utility is often compromised by factual hallucinations and a lack of traceable provenance.<n>We introduce FactNet, a massive, open-source resource designed to unify 1.7 billion atomic assertions with 3.01 billion auditable evidence pointers derived exclusively from 316 Wikipedia editions.
  arXiv  Detail & Related papers  (2026-02-03T11:44:11Z)
• The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems [0.0]
  We apply hallucination prediction to RAG detection, transforming scores into decision sets with finite-sample coverage guarantees.<n>We analyze this failure through the lens of distributional tails, showing that while NLI models achieve acceptable AUC (0.81), the "hardest" hallucinations are semantically indistinguishable from faithful responses.
  arXiv  Detail & Related papers  (2025-12-17T04:22:28Z)
• Citation Failure: Definition, Analysis and Efficient Mitigation [56.09968229868067]
  Citations from LLM-based RAG systems are supposed to simplify response verification.<n>This does not hold for citation failure, when a model generates a helpful response, but fails to cite complete evidence.<n>We propose to disentangle this from response failure, where the response itself is flawed, and citing complete evidence is impossible.
  arXiv  Detail & Related papers  (2025-10-23T07:47:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.