LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents

• URL: http://arxiv.org/abs/2609.09754v1
• Date: Wed, 09 Sep 2026 05:52:53 GMT
• Title: LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents
• Abstract summary: We introduce LexAgentHallu, a legal agentic hallucination benchmark designed to evaluate to what extent and how legal agents fail.<n>Built through a four-stage expert-in-the-loop pipeline, LexAgentHallu contains 3414 instances across 17 legal categories and 6 task types.<n>Our evaluation uncovers a Right-Answer-Wrong-Reason effect and reveals that hallucination subclasses cluster rather than scatter.
• Score: 21.000971972321647
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning errors cascade into fabricated holdings and miscited authority. However, existing legal benchmarks evaluate only single-turn QA with outcome-level metrics, while agentic hallucination benchmarks lack legal-specific diagnostic capability. Neither answers to what extent and how a legal agent hallucinates along its trajectory. To address these limitations, we introduce LexAgentHallu, a legal agentic hallucination benchmark designed to evaluate to what extent and how legal agents fail along multi-step trajectories. Built through a four-stage expert-in-the-loop pipeline, LexAgentHallu contains 3414 instances across 17 legal categories and 6 task types. Each instance is annotated under a dual-layer hallucination taxonomy of 7 high-level categories and 27 fine-grained subclasses, covering both substantive errors and agent-procedural failures. We further design fine-grained metrics that quantify to what extent and localize how each failure occurs along an agent's execution path. Our evaluation across 18 proprietary and open-source agents uncovers a Right-Answer-Wrong-Reason effect and reveals that hallucination subclasses cluster rather than scatter, forming distinct agentic framework, legal task, and category profiles. These findings, invisible to outcome-level evaluation, validate the diagnostic power of LexAgentHallu for evaluating agentic hallucination in law.

Related papers

• Who Checks the Citations? Benchmarking Legal Hallucination Detection [7.592193069681834]
  Attorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently fabricate citations.<n>This study evaluates whether AI-based systems can mitigate these errors by automatically detecting hallucinations.<n>We propose a taxonomy of legal citation hallucinations grounded in actual court filings and introduce a dataset of 1,300 brief excerpts containing injected errors.
  arXiv  Detail & Related papers  (2026-06-19T06:47:31Z)
• LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks [42.88450102623113]
  We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks.<n>It includes 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning.<n>It also includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework.
  arXiv  Detail & Related papers  (2026-06-08T12:04:47Z)
• Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows [1.6048497142263283]
  Trajel is a dataset and evaluation framework for auditing trajectory-level hallucinations in multi-agent agents.<n>We benchmark supervised detection models at the subtask, trajectory, and long-context levels.<n>Trajectory-aware detection significantly outperforms standard post-hoc verification.
  arXiv  Detail & Related papers  (2026-05-22T21:05:10Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents [30.66751974860931]
  hallucination detection in single-turn responses requires identifying which step causes the initial divergence.<n>We propose a new research task, automated hallucination attribution of LLM-based agents, aiming to identify the step responsible for the hallucination and explain why.<n>We introduce AgentHallu, a comprehensive benchmark with 693 high-quality trajectories spanning 7 agent frameworks and 5 domains.<n>The best-performing model achieves only 41.1% step localization accuracy, where tool-use hallucinations are the most challenging at just 11.6%.
  arXiv  Detail & Related papers  (2026-01-11T09:04:26Z)
• Explainable and Fine-Grained Safeguarding of LLM Multi-Agent Systems via Bi-Level Graph Anomaly Detection [76.91230292971115]
  Large language model (LLM)-based multi-agent systems (MAS) have shown strong capabilities in solving complex tasks.<n>XG-Guard is an explainable and fine-grained safeguarding framework for detecting malicious agents in MAS.
  arXiv  Detail & Related papers  (2025-12-21T13:46:36Z)
• LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence [74.05988707492058]
  Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making.<n>Existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs)<n>We propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs.
  arXiv  Detail & Related papers  (2025-12-04T08:48:02Z)
• MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them [52.764019220214344]
  Hallucinations pose critical risks for large language model (LLM)-based agents.<n>We present MIRAGE-Bench, the first unified benchmark for eliciting and evaluating hallucinations in interactive environments.
  arXiv  Detail & Related papers  (2025-07-28T17:38:29Z)
• C-FAITH: A Chinese Fine-Grained Benchmark for Automated Hallucination Evaluation [58.40263551616771]
  We introduce HaluAgent, an agentic framework that automatically constructs fine-grained QA dataset based on some knowledge documents.<n>Our experiments demonstrate that the manually designed rules and prompt optimization can improve the quality of generated data.
  arXiv  Detail & Related papers  (2025-04-14T12:21:55Z)
• LegalAgentBench: Evaluating LLM Agents in Legal Domain [53.70993264644004]
  LegalAgentBench is a benchmark specifically designed to evaluate LLM Agents in the Chinese legal domain.<n>LegalAgentBench includes 17 corpora from real-world legal scenarios and provides 37 tools for interacting with external knowledge.
  arXiv  Detail & Related papers  (2024-12-23T04:02:46Z)
• Gaps or Hallucinations? Gazing into Machine-Generated Legal Analysis for Fine-grained Text Evaluations [38.30926471814935]
  Large Language Models (LLMs) show promise as a writing aid for professionals performing legal analyses. LLMs can often hallucinate in this setting, in ways difficult to recognize by non-professionals and existing text evaluation metrics. We introduce the neutral notion of gaps, as opposed to hallucinations in a strict erroneous sense, to refer to the difference between human-written and machine-generated legal analysis.
  arXiv  Detail & Related papers  (2024-09-16T02:38:38Z)
• Fine-grained Hallucination Detection and Editing for Language Models [109.56911670376932]
  Large language models (LMs) are prone to generate factual errors, which are often called hallucinations. We introduce a comprehensive taxonomy of hallucinations and argue that hallucinations manifest in diverse forms. We propose a novel task of automatic fine-grained hallucination detection and construct a new evaluation benchmark, FavaBench.
  arXiv  Detail & Related papers  (2024-01-12T19:02:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.