SentinelBench: A Benchmark for Long-Running Monitoring Agents

• URL: http://arxiv.org/abs/2606.05342v2
• Date: Fri, 05 Jun 2026 17:19:56 GMT
• Title: SentinelBench: A Benchmark for Long-Running Monitoring Agents
• Abstract summary: SentinelBench is an open-source benchmark for time-evolving monitoring tasks.<n>It contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment.<n>It measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost.
• Score: 20.69666656998877
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: issuing tool calls, refreshing pages, searching for alternatives, or otherwise trying to force progress. This is the wrong approach for many long-running tasks, which are better served by a strategy of sustained attention. Instead, agents should monitor an environment, notice when an external event makes progress possible, then respond promptly without wasting resources while waiting. To measure progress on this class of tasks, we introduce SentinelBench, an open-source benchmark for time-evolving monitoring tasks. SentinelBench contains 100 tasks across 10 synthetic web environments, including email, calendars, finance, professional networking, and entertainment. Each environment exposes a live web interface and replays a scripted sequence of events, requiring agents to navigate and reason about web pages whose state shifts underfoot. SentinelBench measures task completion, reaction time, and resource use, exposing the tradeoff between responsiveness and cost. We report results across three models and two browser-agent harnesses, establishing performance baselines for future comparison and demonstrating how agent design choices can dramatically impact key metrics. Together, these results show that SentinelBench distinguishes meaningful differences in agent behavior.

Related papers

• Agent ATO: Visualizing Agent Interaction Timelines from Logs [1.5639957559514246]
  Agent ATO is a tool for visualizing AI coding agent interaction timelines from console logs.<n>In addition to an all-interaction timeline, Agent ATO provides filtered timelines that emphasize file discovery, file reading, file editing, and execution.
  arXiv  Detail & Related papers  (2026-09-08T06:22:44Z)
• PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents [50.9061759859778]
  PAST-Bench is a benchmark designed to isolate whether retained experience actually improves agents over time.<n>We report both later-task gains and whether those gains follow the intended save, retrieve, and update pathway.<n>We develop Hermes+, which extends Hermes with five targeted interventions across stages of the agent loop.
  arXiv  Detail & Related papers  (2026-08-04T17:58:05Z)
• A Task-State Representation for Long-Horizon Mobile GUI Agents [53.359524744953575]
  Task-State Representation (TSR) is a training-free framework that explicitly decouples task state from sensory input.<n> Experiments across four mobile GUI benchmarks validate TSR's effectiveness, yielding up to a 12 absolute point increase in success rate.
  arXiv  Detail & Related papers  (2026-07-01T06:37:21Z)
• SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? [11.093716663660203]
  SWE-Marathon is a benchmark of 20 long-horizon tasks spanning software engineering and adjacent technical domains.<n>Each task consists of a unique executable environment, a human-written reference solution, and a multi-layer verification suite.<n>Logged agent attempts average 27.2M total tokens, making SWE-Marathon substantially longer-horizon than existing SWE and command-line agent benchmarks.
  arXiv  Detail & Related papers  (2026-06-05T00:39:44Z)
• Multi-Agent Computer Use [72.79887808312706]
  We argue that we should move towards evaluating and building multi-agent computer use (MACU) systems.<n>We propose a general multi-agent setup in which a manager model decomposes computer use tasks as a directed acyclic graph (DAG)<n>At each iteration, the manager dispatches parallel CUA subagents to carry out nodes on the ready frontier of the DAG.
  arXiv  Detail & Related papers  (2026-06-01T01:29:36Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks [67.44219836008348]
  We introduce Odysseys: a benchmark of 200 long-horizon web tasks derived from real world browsing sessions evaluated on the live Internet.<n>We find that binary pass/fail evaluation is inadequate for long-horizon settings and introduce a rubric-based evaluation, annotating each Odysseys task with an average of 6.1 graded rubrics.<n>We find that the strongest models achieve a success rate of 44.5%, which leaves substantial room for future improvements.
  arXiv  Detail & Related papers  (2026-04-27T20:05:41Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• On the Reliability of Computer Use Agents [47.20065484006984]
  We study the sources of unreliability in computer-use agents through three factors.<n> reliability depends on both how tasks are specified and how agent behavior varies across executions.
  arXiv  Detail & Related papers  (2026-04-20T05:59:04Z)
• AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management [24.465443389008055]
  AgentProg is a program-guided approach for agent context management.<n>It reframes the interaction history as a program with variables and control flow.<n> Experiments on AndroidWorld and our extended long-horizon task suite demonstrate that AgentProg has achieved the state-of-the-art success rates.
  arXiv  Detail & Related papers  (2025-12-11T07:37:38Z)
• BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions [48.194688161526756]
  BrowserAgent operates directly on raw web pages via Playwright through a set of predefined browser actions.<n>We introduce an explicit memory mechanism to store key conclusions across steps, further enhancing the model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T15:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.