Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA

• URL: http://arxiv.org/abs/2604.06173v1
• Date: Sat, 24 Jan 2026 06:48:09 GMT
• Title: Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QA
• Abstract summary: We introduce SearchFireSafety, a structure- and safety-aware benchmark for statute-centric legal QA.<n>The benchmark evaluates whether models can retrieve hierarchically fragmented evidence and safely abstain when statutory context is insufficient.
• Score: 8.4220629804904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Legal QA benchmarks have predominantly focused on case law, overlooking the unique challenges of statute-centric regulatory reasoning. In statutory domains, relevant evidence is distributed across hierarchically linked documents, creating a statutory retrieval gap where conventional retrievers fail and models often hallucinate under incomplete context. We introduce SearchFireSafety, a structure- and safety-aware benchmark for statute-centric legal QA. Instantiated on fire-safety regulations as a representative case, the benchmark evaluates whether models can retrieve hierarchically fragmented evidence and safely abstain when statutory context is insufficient. SearchFireSafety adopts a dual-source evaluation framework combining real-world questions that require citation-aware retrieval and synthetic partial-context scenarios that stress-test hallucination and refusal behavior. Experiments across multiple large language models show that graph-guided retrieval substantially improves performance, but also reveal a critical safety trade-off: domain-adapted models are more likely to hallucinate when key statutory evidence is missing. Our findings highlight the need for benchmarks that jointly evaluate hierarchical retrieval and model safety in statute-centric regulatory settings.

Related papers

• Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoning [72.86819994769634]
  Whole-slide image (WSI) reasoning requires an agent to sequentially acquire visual evidence before answering a diagnostic question.<n>We propose BEACON, a plug-and-play agentic framework that reformulates WSI reasoning as a Bayesian evidence acquisition problem.
  arXiv  Detail & Related papers  (2026-08-06T08:40:57Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval [5.305110876082343]
  CaseFacts is a benchmark for verifying legal claims against U.S. Supreme Court precedents.<n>The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled.
  arXiv  Detail & Related papers  (2026-01-23T23:41:46Z)
• Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models [13.70855540464427]
  We introduce Doc-PP, a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies.<n>Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities.<n>We propose DVA, a structural inference framework that decouples reasoning from policy verification.
  arXiv  Detail & Related papers  (2026-01-07T13:45:39Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• An Ontology-Driven Graph RAG for Legal Norms: A Structural, Temporal, and Deterministic Approach [0.0]
  RAG systems in the legal domain face a critical challenge: standard, flat-text retrieval is blind to the hierarchical, diachronic, and causal structure of law, leading to anachronistic and unreliable answers.<n>This paper introduces the Structure-Aware Temporal Graph RAG (SAT-Graph RAG), an ontology-driven framework designed to overcome these limitations by explicitly modeling the formal structure and diachronic nature of legal norms.
  arXiv  Detail & Related papers  (2025-04-29T18:36:57Z)
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
  We introduce DELTA, a discriminative model designed for legal case retrieval. We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Our approach can outperform existing state-of-the-art methods in legal case retrieval.
  arXiv  Detail & Related papers  (2024-03-27T10:40:14Z)
• SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval [75.05173891207214]
  Legal case retrieval plays a core role in the intelligent legal system. Most existing language models have difficulty understanding the long-distance dependencies between different structures. We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
  arXiv  Detail & Related papers  (2023-04-22T10:47:01Z)
• Legal Element-oriented Modeling with Multi-view Contrastive Learning for Legal Case Retrieval [3.909749182759558]
  We propose an interaction-focused network for legal case retrieval with a multi-view contrastive learning objective. Case-view contrastive learning minimizes the hidden space distance between relevant legal case representations. We employ a legal element knowledge-aware indicator to detect legal elements of cases.
  arXiv  Detail & Related papers  (2022-10-11T06:47:23Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.