Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems

• URL: http://arxiv.org/abs/2605.05379v1
• Date: Wed, 06 May 2026 19:01:29 GMT
• Title: Partial Evidence Bench: Benchmarking Authorization-Limited Evidence in Agentic Systems
• Abstract summary: This paper introduces Partial Evidence Bench, a deterministic benchmark for measuring that failure mode.<n>The benchmark ships three scenario families -- due diligence, compliance audit, and security incident response.<n>It evaluates systems along four surfaces: answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Enterprise agents increasingly operate inside scoped retrieval systems, delegated workflows, and policy-constrained evidence environments. In these settings, access control can be enforced correctly while the system still produces an answer that appears complete even though material evidence lies outside the caller's authorization boundary. This paper introduces Partial Evidence Bench, a deterministic benchmark for measuring that failure mode. The benchmark ships three scenario families -- due diligence, compliance audit, and security incident response -- with 72 tasks total, ACL-partitioned corpora, oracle complete answers, oracle authorized-view answers, oracle completeness judgments, and structured gap-report oracles. It evaluates systems along four surfaces: answer correctness, completeness awareness, gap-report quality, and unsafe completeness behavior. Checked-in baselines show that silent filtering is catastrophically unsafe across all shipped families, while explicit fail-and-report behavior eliminates unsafe completeness without collapsing the task into trivial abstention. Preliminary real-model runs show model-dependent and scenario-sensitive differences in whether systems overclaim completeness, conservatively underclaim, or report incompleteness in an enterprise-usable form. The benchmark's broader contribution is to make a governance-critical agent failure measurable without human judges or contamination-prone static corpora.

Related papers

• Cognitive Admission Control: Risk-Conditioned Assurance for Consequential Actions in Agentic Distributed Systems [2.124730017640531]
  In agentic distributed systems, an agent may be authorized to mutate external infrastructure while lacking evidence that the mutation is ready to execute.<n> Cognitive Admission Control (CAC) makes this evidence requirement explicit.<n>A policy maps a typed action and its modeled risk to assurance obligations specifying predicates, evidence classes, scope, freshness, and witness-set constraints.<n>A successful admission produces a certificate binding the action, its witness manifest, and dispatch-time guards.
  arXiv  Detail & Related papers  (2026-09-14T20:20:54Z)
• AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation [48.55515767840701]
  "AI slop" is artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports.<n>This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage.
  arXiv  Detail & Related papers  (2026-08-26T11:48:38Z)
• Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels [53.63220028820581]
  Embodied intelligence integrates learned perception and decision making with real-time computation, control, and physical interaction.<n>We define trustworthy embodied intelligence as the sustained capacity to execute specified tasks reliably.
  arXiv  Detail & Related papers  (2026-07-28T17:50:44Z)
• Task-Conditional Faithfulness Auditing of Multimodal LLMs for Grid Diagnosis [5.427346259545067]
  Multimodal large language models (LLMs) can combine topology, measurements, and incident text for grid diagnosis.<n>This letter proposes a general framework in order to conduct task-conditional faithfulness audit.
  arXiv  Detail & Related papers  (2026-07-27T15:17:15Z)
• A Five-Plane Reference Architecture for Runtime Governance of Production AI Agents [0.0]
  We present a reference architecture for the governance of production AI agents.<n>We define a taxonomy of six interruption primitives that generalize allow and deny, state and argue for four correctness invariants.<n>We demonstrate the foreclosure of seven production-agent threats across five concrete adjudications.
  arXiv  Detail & Related papers  (2026-06-10T16:54:47Z)
• Auditing Agent Harness Safety [81.22315979618612]
  LLM agents increasingly run inside execution harnesses that dispatch tools, allocate resources, and route messages between specialized components.<n>Most safety benchmarks score only final outputs or terminal states, even though many violations occur mid-trajectory rather than at termination.<n>We propose HarnessAudit, a framework that audits full execution trajectories across boundary compliance, execution fidelity, and system stability.
  arXiv  Detail & Related papers  (2026-05-14T02:14:28Z)
• When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents [7.383366380709273]
  Large language model agents increasingly operate through environment-facing scaffolds that expose files, web pages, APIs, and logs.<n>Environmental grounding is a systems-level problem involving context admission, evidence provenance, freshness checking, verification policy, action gating, and model reasoning.<n>We introduce EnvTrustBench, an agentic framework for benchmarking this failure mode.
  arXiv  Detail & Related papers  (2026-05-09T09:32:27Z)
• Governed Auditable Decisioning Under Uncertainty: Synthesis and Agentic Extension [0.0]
  When automated decision systems fail, organizations often discover that formally compliant governance infrastructure cannot reconstruct what happened or why.<n>This paper synthesizes an operational governance evidence framework into an integrated chain and analytically assesses its transferability across four decision system architectures.
  arXiv  Detail & Related papers  (2026-04-21T05:45:04Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Auditable Agents [10.007755929469814]
  LLM agents call tools, query databases, delegate tasks, and trigger external side effects.<n>Once an agent system can act in the world, the question is no longer only whether harmful actions can be prevented--it is whether those actions remain answerable after deployment.<n>We distinguish accountability (the ability to determine compliance and assign responsibility), auditability (the system property that makes accountability possible), and auditing (the process of reconstructing behavior from trustworthy evidence).
  arXiv  Detail & Related papers  (2026-04-07T06:25:49Z)
• Incompleteness of AI Safety Verification via Kolmogorov Complexity [0.20305676256390928]
  We show that limitations of verification arise from intrinsic information-theoretic limits.<n>No finite formal verifier can certify all policy-compliant instances of arbitrarily high complexity.
  arXiv  Detail & Related papers  (2026-04-06T17:26:09Z)
• Agents of Chaos [50.53354213047402]
  We report an exploratory red-teaming study of autonomous language-model-powered agents deployed in a live laboratory environment.<n>Twenty AI researchers interacted with the agents under benign and adversarial conditions.<n>Our findings establish the existence of security-, privacy-, and governance-relevant vulnerabilities in realistic deployment settings.
  arXiv  Detail & Related papers  (2026-02-23T16:28:48Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management [27.25517951457221]
  We introduce DisastQA, a large-scale benchmark of 3,000 rigorously verified questions (2,000 multiple-choice and 1,000 open-ended) spanning eight disaster types.<n>For open-ended QA, we propose a human-verified keypoint-based evaluation protocol emphasizing factual completeness over verbosity.<n>Experiments with 20 models reveal substantial divergences from general-purpose leaderboards such as MMLU-Pro.
  arXiv  Detail & Related papers  (2026-01-07T07:46:42Z)
• Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking [64.97768177044355]
  Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems.<n>We present FactArena, a fully automated arena-style evaluation framework.<n>Our analyses reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence.
  arXiv  Detail & Related papers  (2026-01-06T02:51:56Z)
• SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditions [54.34001921326444]
  Speaker verification (SV) models are increasingly integrated into security, personalization, and access control systems.<n>Existing benchmarks evaluate only subsets of these conditions, missing others entirely.<n>We introduce SVeritas, a comprehensive Speaker Verification tasks benchmark suite, assessing SV systems under stressors like recording duration, spontaneity, content, noise, microphone distance, reverberation, channel mismatches, audio bandwidth, codecs, speaker age, and susceptibility to spoofing and adversarial attacks.
  arXiv  Detail & Related papers  (2025-09-21T14:11:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.