Is there a half-life for the success rates of AI agents?

• URL: http://arxiv.org/abs/2505.05115v1
• Date: Thu, 08 May 2025 10:31:03 GMT
• Title: Is there a half-life for the success rates of AI agents?
• Abstract summary: I show that the performance of AI agents on longer-duration tasks can be explained by a constant rate of failing during each minute a human would take to do the task.<n>This implies an exponentially declining success rate with the length of the task and that each agent could be characterised by its own half-life.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building on the recent empirical work of Kwa et al. (2025), I show that within their suite of research-engineering tasks the performance of AI agents on longer-duration tasks can be explained by an extremely simple mathematical model -- a constant rate of failing during each minute a human would take to do the task. This implies an exponentially declining success rate with the length of the task and that each agent could be characterised by its own half-life. This empirical regularity allows us to estimate the success rate for an agent at different task lengths. And the fact that this model is a good fit for the data is suggestive of the underlying causes of failure on longer tasks -- that they involve increasingly large sets of subtasks where failing any one fails the task. Whether this model applies more generally on other suites of tasks is unknown and an important subject for further work.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• Inducing Task Models from Computer-Use Traces [50.6293197430374]
  Task Model Induction (TMI) discovers latent tasks in an unconstrained trace and disentangling concurrent activity.<n>TMI reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline.<n> skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.
  arXiv  Detail & Related papers  (2026-08-20T17:57:00Z)
• Operationalising the Superficial Alignment Hypothesis via Task Complexity [49.93635747700126]
  We propose a new metric called task complexity: the length of the shortest program that achieves a target performance on a task.<n>Our results highlight that task adaptation often requires surprisingly little information -- often just a few kilobytes.
  arXiv  Detail & Related papers  (2026-02-17T18:59:39Z)
• BRIDGE: Predicting Human Task Completion Time From Model Performance [36.36759710005444]
  Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across benchmarks.<n>We propose BRIDGE, a unified psychometric framework that learns the latent difficulty scale from model responses and anchors it to human task completion time.
  arXiv  Detail & Related papers  (2026-02-06T23:36:11Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks [57.89516354418451]
  We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) We employ a semi-automated task generation pipeline using Large Language Models (LLMs) We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution.
  arXiv  Detail & Related papers  (2024-10-31T17:53:12Z)
• Harnessing Pre-trained Generalist Agents for Software Engineering Tasks [13.733085206098258]
  Deep reinforcement learning (DRL) has been successfully used for automation in complex tasks such as game testing and solving the job-shop scheduling problem. Specialist DRL agents suffer from a lack of generalizability to other tasks and need substantial time to be developed and re-trained effectively. Recently, DRL researchers have begun to develop generalist agents, able to learn a policy from various environments and capable of achieving performances similar to or better than specialist agents in new tasks.
  arXiv  Detail & Related papers  (2023-12-24T18:39:58Z)
• One-shot Visual Imitation via Attributed Waypoints and Demonstration Augmentation [14.391752643263725]
  In one-shot visual imitation, an agent must solve a novel instance of a novel task given just a single visual demonstration. Our analysis reveals that current methods fall short because of three errors: the DAgger problem arising from purely offline training, last centimeter errors in interacting with objects, and mis-fitting to the task context rather than to the actual task. Our model gets 100% and 48% success rates on two recent benchmarks, improving upon the current state-of-the-art by absolute 90% and 20% respectively.
  arXiv  Detail & Related papers  (2023-02-09T18:56:37Z)
• Generalizing to New Tasks via One-Shot Compositional Subgoals [23.15624959305799]
  The ability to generalize to previously unseen tasks with little to no supervision is a key challenge in modern machine learning research. We introduce CASE which attempts to address these issues by training an Imitation Learning agent using adaptive "near future" subgoals. Our experiments show that the proposed approach consistently outperforms the previous state-of-the-art compositional Imitation Learning approach by 30%.
  arXiv  Detail & Related papers  (2022-05-16T14:30:11Z)
• Conflict-Averse Gradient Descent for Multi-task Learning [56.379937772617]
  A major challenge in optimizing a multi-task model is the conflicting gradients. We introduce Conflict-Averse Gradient descent (CAGrad) which minimizes the average loss function. CAGrad balances the objectives automatically and still provably converges to a minimum over the average loss.
  arXiv  Detail & Related papers  (2021-10-26T22:03:51Z)
• Efficiently Identifying Task Groupings for Multi-Task Learning [55.80489920205404]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single training run by co-training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss.
  arXiv  Detail & Related papers  (2021-09-10T02:01:43Z)
• Hierarchical Few-Shot Imitation with Skill Transition Models [66.81252581083199]
  Few-shot Imitation with Skill Transition Models (FIST) is an algorithm that extracts skills from offline data and utilizes them to generalize to unseen tasks. We show that FIST is capable of generalizing to new tasks and substantially outperforms prior baselines in navigation experiments.
  arXiv  Detail & Related papers  (2021-07-19T15:56:01Z)
• Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Learning [80.66108902283388]
  Multi-task learning methods should be used with caution for safety-critical applications, such as clinical risk prediction. Existing asymmetric multi-task learning methods tackle this negative transfer problem by performing knowledge transfer from tasks with low loss to tasks with high loss. We propose a novel temporal asymmetric multi-task learning model that performs knowledge transfer from certain tasks/timesteps to relevant uncertain tasks, based on feature-level uncertainty.
  arXiv  Detail & Related papers  (2020-06-23T06:01:36Z)
• Hierarchical Reinforcement Learning as a Model of Human Task Interleaving [60.95424607008241]
  We develop a hierarchical model of supervisory control driven by reinforcement learning. The model reproduces known empirical effects of task interleaving. The results support hierarchical RL as a plausible model of task interleaving.
  arXiv  Detail & Related papers  (2020-01-04T17:53:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.