Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents

• URL: http://arxiv.org/abs/2603.29231v1
• Date: Tue, 31 Mar 2026 03:56:39 GMT
• Title: Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents
• Abstract summary: Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability.<n>We introduce a reliability science framework for long-horizon LLM agents with four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP)<n>We evaluate 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing benchmarks measure capability -- whether a model succeeds on a single attempt -- but production deployments require reliability -- consistent success across repeated attempts on tasks of varying duration. We show these properties diverge systematically as task duration grows, and that pass@1 on short tasks is structurally blind to this divergence. We introduce a reliability science framework for long-horizon LLM agents with four metrics: Reliability Decay Curve (RDC), Variance Amplification Factor (VAF), Graceful Degradation Score (GDS), and Meltdown Onset Point (MOP). We evaluate 10 models across 23,392 episodes on a 396-task benchmark spanning four duration buckets and three domains. Key findings: (1) reliability decay is domain-stratified -- SE GDS drops from 0.90 to 0.44 while document processing is nearly flat (0.74 to 0.71); (2) VAF bifurcates by capability tier -- high VAF is a capability signature, not an instability signal; (3) capability and reliability rankings diverge substantially, with multi-rank inversions at long horizons; (4) frontier models have the highest meltdown rates (up to 19%) because they attempt ambitious multi-step strategies that sometimes spiral; and (5) memory scaffolds universally hurt long-horizon performance across all 10 models. These results motivate reliability as a first-class evaluation dimension alongside capability.

Related papers

• How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making [0.0]
  Production deployments of large language model (LLM) agents remain unreliable on long, multi-step even as benchmark success rates climb steadily.<n>We measure the effect directly, characterizing the shape of agent degradation and disentangling its cause.<n>For teams responsible for agent orchestration and reliability at scale, these results argue for horizon-aware evaluation and reliability budgeting.
  arXiv  Detail & Related papers  (2026-08-31T19:11:47Z)
• Unveiling the Entropy Dynamics of Chain-of-Thought Reasoning [81.57028614960576]
  We show that the Confidence Region possesses two critical properties: 1) High Reliability -- answers in the confidence region become highly accurate and stable, and 2) High Redundancy -- models generate unnecessary tokens long after reaching the correct answer.<n>These properties unlock more efficient and reliable inference strategies.
  arXiv  Detail & Related papers  (2026-06-01T10:11:14Z)
• Hallucination Mitigation with Agentic AI, Nested Learning, and AI Sustainability via Semantic Caching [0.42970700836450487]
  This paper adapts a HOPE-inspired Nested Learning architecture with Continuum Memory Systems (CMS)<n>A three-stage agentic pipeline is evaluated with fiveaug -- FCDFactual Claim Density, FGR, THS (Total Hallucination Score) and OSR (Observability Score Ratio)<n>Semantic caching achieves 440 cache hits over 930 potential calls (47.3% hit rate), reducing LLM invocations to 490, lowering energy and CO2e footprint, and making multi-stage review pipelines operationally viable at production scale.
  arXiv  Detail & Related papers  (2026-05-27T20:01:08Z)
• Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks [45.86413490112477]
  Large language models (LLMs) are used in reliability-sensitive applications.<n>Estimateing rare failure probability with tight confidence bounds requires prohibitively large LLM inference sizes.<n>We propose to learn a sampling distribution concentrated on failure-prone inputs via the cross-entropy method.
  arXiv  Detail & Related papers  (2026-05-11T20:23:44Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Multi-Axis Trust Modeling for Interpretable Account Hijacking Detection [1.0152838128195467]
  This paper proposes a Hadith-inspired multi-axis trust modeling framework.<n>We translate five trust axes - long-term integrity (adalah), behavioral precision (dabt), contextual continuity (isnad), cumulative reputation, and anomaly evidence - into a compact set of 26 semantically meaningful behavioral features for user accounts.<n>We evaluate the framework on the CLUE-LDS cloud activity dataset with injected account hijacking scenarios.
  arXiv  Detail & Related papers  (2026-02-20T19:36:30Z)
• BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning [73.46118996284888]
  Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence.<n>We propose BadCLIP++, a unified framework that tackles both challenges.<n>For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions.<n>For persistence, we stabilize trigger embeddings via radius shrinkage and centroid alignment.
  arXiv  Detail & Related papers  (2026-02-19T08:31:16Z)
• The Six Sigma Agent: Achieving Enterprise-Grade Reliability in LLM Systems Through Consensus-Driven Decomposed Execution [0.0]
  We introduce the Six Sigma Agent, a novel architecture that achieves enterprise-grade reliability through three synergistic components.<n>We demonstrate a 14,700x reliability improvement over single-agent execution while reducing costs by 80%.<n>Our work establishes that reliability in AI systems emerges from principled redundancy and consensus rather than model scaling alone.
  arXiv  Detail & Related papers  (2026-01-29T20:04:29Z)
• ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions [0.32928123659012326]
  Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production.<n>We introduce textbfReliabilityBench, a benchmark for evaluating agent reliability across three dimensions.<n>We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes.
  arXiv  Detail & Related papers  (2026-01-03T13:41:33Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• ReFIne: A Framework for Trustworthy Large Reasoning Models with Reliability, Faithfulness, and Interpretability [23.70973331911138]
  We argue that usable reasoning systems must be trustworthy, characterized by three properties: interpretability, faithfulness, and reliability.<n>We propose ReFIne, a new training framework that integrates supervised fine-tuning with GRPO to encourage models to improve interpretability.<n>Our experimental results show that ReFIne models generate clearer and better-structured reasoning traces.
  arXiv  Detail & Related papers  (2025-10-10T07:08:44Z)
• On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations [52.1029745126386]
  In vision-language-action (VLA) models, robustness to real-world perturbations is critical for deployment.<n>We propose RobustVLA against perturbations in VLA inputs and outputs.<n> Experiments on LIBERO demonstrate our RobustVLA delivers absolute gains over baselines of 12.6% on the pi0 backbone and 10.4% on the OpenVLA backbone.
  arXiv  Detail & Related papers  (2025-09-26T14:42:23Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Confidence-Aware Routing for Large Language Model Reliability Enhancement: A Multi-Signal Approach to Pre-Generation Hallucination Mitigation [0.0]
  Large Language Models suffer from hallucination, generating plausible yet factually incorrect content.<n>Current mitigation strategies focus on post-generation correction, which is computationally expensive and fails to prevent unreliable content generation.<n>We propose a confidence-aware routing system that proactively assesses model uncertainty before generation and redirects queries based on estimated reliability.
  arXiv  Detail & Related papers  (2025-09-23T18:34:20Z)
• Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression [109.23761449840222]
  This study conducts the first, thorough evaluation of leading Large Language Models (LLMs) We find that quantization is currently a more effective approach than pruning in achieving efficiency and trustworthiness simultaneously.
  arXiv  Detail & Related papers  (2024-03-18T01:38:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.