READY or Not: Reliable Enterprise Agent Deployment

• URL: http://arxiv.org/abs/2609.02095v1
• Date: Wed, 02 Sep 2026 04:34:48 GMT
• Title: READY or Not: Reliable Enterprise Agent Deployment
• Abstract summary: READY is a framework for qualifying AI agents for deployment on enterprise.<n>It measures the reliability and operating cost of the human-AI system.<n>It statistically qualifies it on held-out cases.
• Score: 16.430548759635037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, whereas enterprise deployment asks a different question: whether an agent can meet a required reliability level, under acceptable human oversight, and at tolerable cost. We introduce Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows. READY preserves each workflow's own definition of successful execution while applying a common qualification procedure. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human-AI system, selects the minimum-cost policy that satisfies a specified reliability target, and statistically qualifies it on held-out cases. The resulting deployment profile characterizes the supported operating point: reliability, human-oversight burden, and cost. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and qualification, and runs on existing agent-evaluation infrastructure. In an end-to-end clinical-audit case study spanning 16 agent systems and 750 cases, READY reveals differences hidden by autonomous performance: two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy. READY thus shifts enterprise agent evaluation from how well can the agent perform the work? to under what conditions, and at what cost, can it be reliably deployed? By making those conditions explicit and statistically testable, READY provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.

Related papers

• A Policy Algebra for Trust-Preserving Agentic AI Execution [5.943264605557423]
  We propose a policy algebra that defines the reliability envelope within which agent capability may be exercised.<n>The evaluation is interpreted as a reliability-capability trade-off rather than a capability benchmark.
  arXiv  Detail & Related papers  (2026-08-17T10:59:15Z)
• Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness [0.0]
  This paper introduces the ProofAgent Index (PAI), a governance readiness index for AI agents.<n>PAI combines four dimensions of deployment evidence: Evaluation, Context, Compliance, and Governance.<n>It is implemented inside ProofAgent Harness, an open source infrastructure for auditable AI agent evaluation and governance.
  arXiv  Detail & Related papers  (2026-07-30T04:44:47Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End? [28.383940617377856]
  We introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on operations research tasks.<n>It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema.<n>Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice.
  arXiv  Detail & Related papers  (2026-06-18T04:43:23Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? [150.84850629123287]
  End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks.<n>$$-Bench is a benchmark of long-horizon healthcare across three domains: provider prior authorization, payer utilization management, and care management.<n>Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass3, and executing all tasks in a single session slumps the performance to 3.8%.
  arXiv  Detail & Related papers  (2026-05-15T22:34:31Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems [0.0]
  Current agentic AI benchmarks predominantly evaluate task completion accuracy.<n>Lack of cost-controlled evaluation leads to 50x cost variations for similar precision.<n>Inadequate reliability assessment where agent performance drops from 60% (single run) to 25% (8-run consistency)
  arXiv  Detail & Related papers  (2025-11-18T04:50:19Z)
• OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety [58.201189860217724]
  We introduce OpenAgentSafety, a comprehensive framework for evaluating agent behavior across eight critical risk categories.<n>Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms.<n>It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.
  arXiv  Detail & Related papers  (2025-07-08T16:18:54Z)
• AI Agents-as-Judge: Automated Assessment of Accuracy, Consistency, Completeness and Clarity for Enterprise Documents [0.0]
  This study presents a modular, multi-agent system for the automated review of highly structured enterprise business documents using AI agents.<n>It uses modern orchestration tools such as LangChain, CrewAI, TruLens, and Guidance to enable section-by-section evaluation of documents.<n>It achieves 99% information consistency (vs. 92% for humans), halving error and bias rates, and reducing average review time from 30 to 2.5 minutes per document.
  arXiv  Detail & Related papers  (2025-06-23T17:46:15Z)
• Agent-as-a-Judge: Evaluate Agents with Agents [61.33974108405561]
  We introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We present DevAI, a new benchmark of 55 realistic automated AI development tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.