E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing

• URL: http://arxiv.org/abs/2512.03109v1
• Date: Tue, 02 Dec 2025 05:59:18 GMT
• Title: E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing
• Abstract summary: We introduce e-valuator, a method to convert any black-box verifier score into a decision rule with provable control of false alarm rates.<n>Evaluator builds on tools from eprocesses to develop a sequential hypothesis test that remains statistically valid at every step of an agent's trajectory.<n>We show that e-valuator provides greater statistical power and better false alarm rate control than other strategies across six datasets and three agents.
• Score: 7.984665398116918
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic AI systems execute a sequence of actions, such as reasoning steps or tool calls, in response to a user prompt. To evaluate the success of their trajectories, researchers have developed verifiers, such as LLM judges and process-reward models, to score the quality of each action in an agent's trajectory. Although these heuristic scores can be informative, there are no guarantees of correctness when used to decide whether an agent will yield a successful output. Here, we introduce e-valuator, a method to convert any black-box verifier score into a decision rule with provable control of false alarm rates. We frame the problem of distinguishing successful trajectories (that is, a sequence of actions that will lead to a correct response to the user's prompt) and unsuccessful trajectories as a sequential hypothesis testing problem. E-valuator builds on tools from e-processes to develop a sequential hypothesis test that remains statistically valid at every step of an agent's trajectory, enabling online monitoring of agents over arbitrarily long sequences of actions. Empirically, we demonstrate that e-valuator provides greater statistical power and better false alarm rate control than other strategies across six datasets and three agents. We additionally show that e-valuator can be used for to quickly terminate problematic trajectories and save tokens. Together, e-valuator provides a lightweight, model-agnostic framework that converts verifier heuristics into decisions rules with statistical guarantees, enabling the deployment of more reliable agentic systems.

Related papers

• HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution [11.431479238875712]
  We introduce HarnessEvolve, a self-evolving framework that learns from reference trajectories to achieve reliable agent self-evolution.<n>We conduct extensive experiments on several benchmarks spanning open-domain and enterprise scenarios, using different models and agent frameworks.<n>Results demonstrate that HarnessEvolve consistently outperforms state-of-the-art baselines across all benchmarks and settings.
  arXiv  Detail & Related papers  (2026-09-01T07:31:18Z)
• Towards Risk-free AI Agent Deployment [13.502827903528948]
  We argue that risk-free deployment must be grounded in the agent's trajectory.<n>This article begins with the challenges of testing agents, including the oracle problem.<n>We distill these directions into a practical deployment-readiness checklist.
  arXiv  Detail & Related papers  (2026-08-17T11:07:07Z)
• From Sequence to Structure: Relational Uncertainty Propagation for LLM Agents [75.69180947909148]
  We propose a trajectory-level UQ framework for large language model (LLM) agents.<n>RuPA represents an execution history as a directed trajectory graph in which reasoning states, tool interactions, and environment feedback are nodes connected by temporal and semantic dependency edges.<n>We evaluate RUPA on representative agent benchmarks, including $$-2, Terminal-Bench-2, and GAIA, using 6 open-source LLMs spanning multiple model families.
  arXiv  Detail & Related papers  (2026-08-17T01:40:14Z)
• Controllable Sim Agents with Behavior Latents [9.41175594126692]
  Controllable Variational Agents (CNeVA) is a controllable simulated-agent framework.<n>Speed- and acceleration-based steering produces monotone responses without stall-induced reward hacking.<n>Our experiment demonstrates that steering metrics must be read alongside physical-plausibility guardrails to avoid reward-hacking confounds.
  arXiv  Detail & Related papers  (2026-07-02T17:55:39Z)
• ToolChain-CRC: Conformal Risk Control for Agentic AI Under Retrieval and Tool-Use Drift [0.0]
  We propose ToolChain-CRC, a conformal risk-control method for retrieval-augmented and tool-using agents under drift.<n>The method treats each agent run as a full trajectory of actions, observations, and final output.<n>It builds step-level risk scores, combines them into a trajectory risk score, calibrates an accept-or-intervene rule, and adds an anytime alarm.
  arXiv  Detail & Related papers  (2026-06-16T20:27:37Z)
• VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems [79.51005192758262]
  Large language model-driven multi-agent systems excel at complex tasks.<n>However, unreliable agents remain a key bottleneck to system-level reliability.<n>We propose a hypothesis verification framework for agent failure attribution.
  arXiv  Detail & Related papers  (2026-05-17T14:09:35Z)
• AgentV-RL: Scaling Reward Modeling with Agentic Verifier [63.55502685076245]
  Verifiers have been demonstrated to enhance LLM reasoning via test-time scaling (TTS)<n>We propose Agentic Verifier, a framework that transforms reward modeling into a multi-turn, tool-augmented deliberative process.<n>We show that Agentic Verifier yields consistent performance gains under both parallel and sequential TTS.
  arXiv  Detail & Related papers  (2026-04-17T12:27:36Z)
• Detecting Object Tracking Failure via Sequential Hypothesis Testing [80.7891291021747]
  Real-time online object tracking in videos constitutes a core task in computer vision.<n>We propose interpreting object tracking as a sequential hypothesis test, wherein evidence for or against tracking failures is gradually accumulated over time.<n>We propose both supervised and unsupervised variants by leveraging either ground-truth or solely internal tracking information.
  arXiv  Detail & Related papers  (2026-02-13T14:57:15Z)
• Scaling Agentic Verifier for Competitive Coding [66.11758166379092]
  Large language models (LLMs) have demonstrated strong coding capabilities but still struggle to solve competitive programming problems correctly in a single attempt.<n>Execution-based re-ranking offers a promising test-time scaling strategy, yet existing methods are constrained by either difficult test case generation or inefficient random input sampling.<n>We propose Agentic Verifier, an execution-based agent that actively reasons about program behaviors and searches for highly discriminative test inputs.
  arXiv  Detail & Related papers  (2026-02-04T06:30:40Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack [52.17935054046577]
  We present ReasAlign, a model-level solution to improve safety alignment against indirect prompt injection attacks.<n>ReasAlign incorporates structured reasoning steps to analyze user queries, detect conflicting instructions, and preserve the continuity of the user's intended tasks.
  arXiv  Detail & Related papers  (2026-01-15T08:23:38Z)
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction [58.51530390018909]
  Large Language Model based multi-agent systems excel at collaborative problem solving but remain brittle to cascading errors.<n>We present MASC, a metacognitive framework that endows MAS with real-time, unsupervised, step-level error detection and self-correction.
  arXiv  Detail & Related papers  (2025-10-16T05:35:37Z)
• Towards Self-Evolving Benchmarks: Synthesizing Agent Trajectories via Test-Time Exploration under Validate-by-Reproduce Paradigm [60.36837655498119]
  We propose a Trajectory-based validated-by-Reproducing Agent-benchmark Complexity Evolution framework.<n>This framework takes an original task from an existing benchmark and encourages agents to evolve it into a new task with higher difficulty.<n>Experiments on the GAIA benchmark demonstrate that the TRACE framework consistently enhances task complexity while improving the reliability of correctness.
  arXiv  Detail & Related papers  (2025-10-01T01:52:52Z)
• AgentGuard: Runtime Verification of AI Agents [1.14219428942199]
  AgentGuard is a framework for runtime verification of Agentic AI systems.<n>It provides continuous, quantitative assurance through a new paradigm called Dynamic Probabilistic Assurance.
  arXiv  Detail & Related papers  (2025-09-28T13:08:50Z)
• Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference [8.823529310904162]
  Multi-agent systems (MAS) are critical for automating complex tasks, yet their practical deployment is hampered by the challenge of failure attribution.<n>We introduce the first failure attribution framework for MAS grounded in multi-granularity causal inference.
  arXiv  Detail & Related papers  (2025-09-10T15:22:00Z)
• Towards Robust Fact-Checking: A Multi-Agent System with Advanced Evidence Retrieval [1.515687944002438]
  The rapid spread of misinformation in the digital era poses significant challenges to public discourse.<n>Traditional human-led fact-checking methods, while credible, struggle with the volume and velocity of online content.<n>This paper proposes a novel multi-agent system for automated fact-checking that enhances accuracy, efficiency, and explainability.
  arXiv  Detail & Related papers  (2025-06-22T02:39:27Z)
• Active Test-time Vision-Language Navigation [60.69722522420299]
  ATENA is a test-time active learning framework that enables a practical human-robot interaction via episodic feedback on uncertain navigation outcomes.<n>In particular, ATENA learns to increase certainty in successful episodes and decrease it in failed ones, improving uncertainty calibration.<n>In addition, we propose a self-active learning strategy that enables an agent to evaluate its navigation outcomes based on confident predictions.
  arXiv  Detail & Related papers  (2025-06-07T02:24:44Z)
• Explaining Reinforcement Learning Policies through Counterfactual Trajectories [147.7246109100945]
  A human developer must validate that an RL agent will perform well at test-time. Our method conveys how the agent performs under distribution shifts by showing the agent's behavior across a wider trajectory distribution. In a user study, we demonstrate that our method enables users to score better than baseline methods on one of two agent validation tasks.
  arXiv  Detail & Related papers  (2022-01-29T00:52:37Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.