LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI

• URL: http://arxiv.org/abs/2606.18021v1
• Date: Tue, 16 Jun 2026 15:02:37 GMT
• Title: LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
• Abstract summary: LegalHalluLens is an auditing framework for legal AI deployed in the wild.<n>Typed profiles and RDI surface failure modes that aggregate metrics hide.<n>The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI systems deployed in legal workflows hallucinate at rates that aggregate metrics report at ~52%, but this average conceals where errors concentrate and in which direction they run, leaving compliance officers without an actionable signal for trustworthy deployment. We present LegalHalluLens, an auditing framework with three components: typed hallucination profiles across four legally-motivated claim categories (numeric, temporal, obligation/entitlement, factual) over CUAD (Hendrycks et al., 2021); a Risk Direction Index (RDI) that reduces omission-versus-invention bias to a single deployment-comparable scalar; and a typed debate pipeline calibrated to both magnitudes and directions. Across 510 contracts and 249,252 clause-level instances we measure a within-model gap of approximately 38-40 pp between obligation/numeric and temporal claims that aggregate reporting hides, and show that two systems with matched 52% rates can carry opposite RDIs. The debate pipeline reduces fabricated detections by 45% with per-category gains tracking the diagnosis, matching commercial APIs with a substantially smaller backbone (4B active parameters). Typed profiles and RDI surface failure modes that aggregate metrics hide; we further show these diagnostics serve as calibration inputs for multi-agent debate pipelines, where Skeptic challenges and asymmetric gates targeted at measured failure modes outperform generically-tuned debate. The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.

Related papers

• Who Bears the Risk When Generative AI Enters Transport? A Distributional Sociotechnical Audit of Algorithmic Equity, Synthetic-Data Validity, and Public Trust [1.089614199781423]
  Generative artificial intelligence is entering transportation through traveler-facing advisories, synthetic crash-record generation, and policy decision support.<n>We develop a Distributional Audit that integrates algorithmic equity, synthetic-data validity, and public-attitude into one empirical pipeline.
  arXiv  Detail & Related papers  (2026-09-10T14:29:09Z)
• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation [48.55515767840701]
  "AI slop" is artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports.<n>This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage.
  arXiv  Detail & Related papers  (2026-08-26T11:48:38Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Cascading Hallucination in Agentic RAG: The CHARM Framework for Detection and Mitigation [0.0]
  We introduce CHARM, an architectural framework for detecting and interrupting error propagation in multi-step reasoning pipelines.<n> CHARM comprises four components - stage-level fact verification, cross-stage consistency tracking, confidence propagation monitoring, and cascade resolution triggering.<n>We evaluate CHARM on HotpotQA, MuSiQue, 2WikiMultiHopQA, and a custom adversarial dataset across LangChain agentic pipeline configurations.
  arXiv  Detail & Related papers  (2026-06-03T04:33:47Z)
• Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring [0.0]
  Current approaches to AI compliance treat conformity as a binary, audit-time verdict.<n>We argue that this compliance fiction is structurally ill-suited to the requirements of the EU AI Act.<n>We introduce governance from metrics, a principle whereby regulatory compliance is derived as a continuous signal from behavioural observability.
  arXiv  Detail & Related papers  (2026-05-23T21:21:33Z)
• ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling [57.42714978834704]
  ExComm is a communication protocol for exploration-stage agentic test-time scaling.<n>We show that ExComm consistently outperforms strong test-time scaling baselines.
  arXiv  Detail & Related papers  (2026-05-21T07:38:44Z)
• GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs [0.0]
  We present a groundedness framework that partitions claims into a four-way typology (grounded, ungrounded, contradicted, complementary)<n>GSAR is the first published groundedness framework coupling evidence-typed scoring with tiered recovery under an explicit compute budget.
  arXiv  Detail & Related papers  (2026-04-25T16:20:28Z)
• ArgRE: Formal Argumentation for Conflict Resolution in Multi-Agent Requirements Negotiation [3.512547713134864]
  ArgRE is a multi-agent requirements system that embeds abstract argumentation into the negotiation stage.<n>It provides argument-level traceability absent from existing frameworks.<n> Evaluation shows that ArgRE provides argument-level traceability absent from existing frameworks.
  arXiv  Detail & Related papers  (2026-04-25T03:28:13Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learning [46.294745464571456]
  LongCat-Flash-Prover is an open-source MoE model for agentic tool-integrated reasoning.<n>It sets a new state-of-the-art for open-weights models in both auto-formalization and theorem proving.<n>It achieves a 97.1% pass rate on MiniF2F-Test using only 72 inference budget per problem.
  arXiv  Detail & Related papers  (2026-03-22T05:16:09Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG [5.4716896485317195]
  Current stateless defences for multimodal agentic RAG fail to detect adversarial strategies that distribute malicious semantics.<n>We introduce MMA-RAGT, an inference-time control framework governed by a Modular Trust Agent (MTA) that maintains an approximate belief state.
  arXiv  Detail & Related papers  (2026-02-24T23:52:27Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses.<n>In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources.<n>We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.