InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

• URL: http://arxiv.org/abs/2608.20220v1
• Date: Thu, 20 Aug 2026 16:14:47 GMT
• Title: InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
• Abstract summary: We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency.<n>We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes.<n>No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.
• Score: 18.0624065854059
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.

Related papers

• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement [0.573978744524696]
  We study how a model manages its authority, referring here to its claim to knowledge, competence, or the right to advise, once a user challenges its answer.<n>We construct a new dataset of 2,310 controlled challenge scenarios and 32,340 corresponding responses from 14 models.<n>We find that models show conflicting behaviour: they validate users in 85% of responses but maintain their original claim in 65%.
  arXiv  Detail & Related papers  (2026-09-07T15:47:29Z)
• LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks [42.88450102623113]
  We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks.<n>It includes 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning.<n>It also includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework.
  arXiv  Detail & Related papers  (2026-06-08T12:04:47Z)
• CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval [5.305110876082343]
  CaseFacts is a benchmark for verifying legal claims against U.S. Supreme Court precedents.<n>The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled.
  arXiv  Detail & Related papers  (2026-01-23T23:41:46Z)
• AppellateGen: A Benchmark for Appellate Legal Judgment Generation [30.9030336647868]
  We introduce AppellateGen, a benchmark for second-instance legal judgment generation comprising 7,351 case pairs.<n>The task requires models to draft legally binding judgments by reasoning over the initial verdict and evidentiary updates.<n>We propose a judicial Standard Operating Procedure (SOP)-based Legal Multi-Agent System (SLMAS) to simulate judicial, which decomposes the generation process into discrete stages of issue identification, retrieval, and drafting.
  arXiv  Detail & Related papers  (2026-01-04T02:15:17Z)
• Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements [78.87065404966002]
  Existing benchmarks predominantly curate questions at the question level.<n>We propose Encyclo-K, a statement-based benchmark that rethinks benchmark construction from the ground up.
  arXiv  Detail & Related papers  (2025-12-31T13:55:54Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation [56.79698529022327]
  Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution.<n>This paper explores the problem of legal claim generation based on the given case's facts.<n>We construct ClaimGen-CN, the first dataset for Chinese legal claim generation task.
  arXiv  Detail & Related papers  (2025-08-24T07:19:25Z)
• LegalSearchLM: Rethinking Legal Case Retrieval as Legal Elements Generation [9.894351313663874]
  Legal Case Retrieval (LCR) is a fundamental task for legal professionals.<n>Existing studies on LCR face two major limitations.<n>First, they are evaluated on relatively small-scale retrieval corpora.<n>Second, their reliance on embedding-based or lexical matching methods often results in limited representations and legally irrelevant matches.
  arXiv  Detail & Related papers  (2025-05-28T09:02:41Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
  We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• The Art of Saying No: Contextual Noncompliance in Language Models [123.383993700586]
  We introduce a comprehensive taxonomy of contextual noncompliance describing when and how models should not comply with user requests. Our taxonomy spans a wide range of categories including incomplete, unsupported, indeterminate, and humanizing requests. To test noncompliance capabilities of language models, we use this taxonomy to develop a new evaluation suite of 1000 noncompliance prompts.
  arXiv  Detail & Related papers  (2024-07-02T07:12:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.