The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?

• URL: http://arxiv.org/abs/2609.08589v1
• Date: Tue, 08 Sep 2026 11:25:31 GMT
• Title: The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?
• Abstract summary: Recent large language models can emit task-progress signals that agent frameworks use to decide whether a task should continue or stop.<n>We evaluate this ability on the public benchmark $2$-bench and on StageIF, a controlled testbed in which reporting checkpoints are placed across the task's lifecycle.<n>We find that reporting reliability depends on the stage a task has reached, and that almost every deployed model we test is reliable at some stages and unreliable at others.
• Score: 9.023267560784042
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent large language models can emit task-progress signals that agent frameworks use to decide whether a task should continue or stop, yet whether a model can reliably report its task progress at every stage of a task, and where and how its reports fail, has not been studied systematically. We evaluate this ability on the public benchmark $τ^2$-bench and on StageIF, a controlled testbed in which reporting checkpoints are placed across the task's lifecycle. Both settings require reports at multiple task stages. We find that reporting reliability depends on the stage a task has reached, and that almost every deployed model we test is reliable at some stages and unreliable at others. Where reporting breaks down is not the same everywhere. Most deployed models lose accuracy once work is under way and recover once the task is done. The newest generation closes that mid-task drop and instead grows conservative at the finish line. Our study exposes a capability gap in task-progress reporting and provides an evaluation protocol that spans the whole course of task execution for this ability on which agent operation depends. The findings indicate that agent frameworks should not control task flow on the strength of the model's state reports alone.

Related papers

• AgentRewind: Recoverable Execution for Long-Horizon LLM Agents [32.44158037765297]
  We present AgentRewind, a runtime recovery framework that records aligned checkpoints of the agent context and controlled environment.<n>We also construct MettleBench, a benchmark for evaluating task completion and partial progress on long-horizon engineering assignments.
  arXiv  Detail & Related papers  (2026-08-14T15:20:35Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers [0.0]
  We present DynamicMCPBench, a reusable framework rather than a fixed dataset.<n>It generates realistic goals, pursues each one live to record a successful trajectory, distills that trajectory into path-agnostic effect checkpoints, and scores an agent on whether it reproduces those effects, never on the final answer.<n>Even the strongest agents solve only about half of the tasks, 31% of tasks are solved by no model at all, and accuracy collapses as the required tool chain grows longer.
  arXiv  Detail & Related papers  (2026-07-10T12:25:57Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• On the Reliability of Computer Use Agents [47.20065484006984]
  We study the sources of unreliability in computer-use agents through three factors.<n> reliability depends on both how tasks are specified and how agent behavior varies across executions.
  arXiv  Detail & Related papers  (2026-04-20T05:59:04Z)
• Learning Stable Classifiers by Transferring Unstable Features [59.06169363181417]
  We study transfer learning in the presence of spurious correlations. We experimentally demonstrate that directly transferring the stable feature extractor learned on the source task may not eliminate these biases for the target task. We hypothesize that the unstable features in the source task and those in the target task are directly related.
  arXiv  Detail & Related papers  (2021-06-15T02:41:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.