Predicting Task Difficulty Without Rollouts

• URL: http://arxiv.org/abs/2608.05797v1
• Date: Thu, 06 Aug 2026 09:33:09 GMT
• Title: Predicting Task Difficulty Without Rollouts
• Abstract summary: We study textitex ante difficulty prediction across 17 agentic benchmarks spanning coding, mathematics, machine learning, web navigation, function calling, and other domains.<n>We show that AUC can mask poor difficulty estimates, identify token-level entropy as a useful predictive signal, and show how residuals between expected and observed difficulty can expose hidden environment flaws such as contamination and infeasibility.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments. Reliable estimates would therefore allow environment designers to calibrate evaluation benchmarks and construct progressive training curricula. This becomes increasingly important as agents move into long-horizon domains, where empirical trial-and-error is a severe computational bottleneck. Prior work on early prediction is limited to static tasks or isolated coding environments, often relying on narrow features and inaccurate evaluation metrics. We study \textit{ex ante} difficulty prediction across 17 agentic benchmarks spanning coding, mathematics, machine learning, web navigation, function calling, and other domains. We show that AUC can mask poor difficulty estimates, identify token-level entropy as a useful predictive signal, and show how residuals between expected and observed difficulty can expose hidden environment flaws such as contamination and infeasibility.

Related papers

• What Makes Software Issue Resolution Tasks Difficult for Agents? [5.974359772961166]
  We investigate what structural properties of software tasks correspond to agent success rates for issue resolution tasks.<n>We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date.
  arXiv  Detail & Related papers  (2026-08-18T19:59:16Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Modeling Sparse and Bursty Vulnerability Sightings: Forecasting Under Data Constraints [46.07145155913717]
  vulnerability sightings, such as proof-of-concept releases, detection templates, or online discussions, can be forecast over time.<n>We evaluate several approaches for short-term forecasting of sightings per vulnerability.<n>Count-based methods such as Poisson regression produce more stable and interpretable forecasts.
  arXiv  Detail & Related papers  (2026-04-17T13:06:46Z)
• Preliminary Investigation into Uncertainty-Aware Attack Stage Classification [81.28215542218724]
  This work addresses the problem of attack stage inference under uncertainty.<n>We propose a classification approach based on Evidential Deep Learning (EDL), which models predictive uncertainty by outputting parameters of a Dirichlet distribution over possible stages.<n>Preliminary experiments in a simulated environment demonstrate that the proposed model can accurately infer the stage of an attack with confidence.
  arXiv  Detail & Related papers  (2025-08-01T06:58:00Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably.<n>This poses a significant challenge to ensuring their safe deployment.<n>We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• Model Predictive Task Sampling for Efficient and Robust Adaptation [57.414812940406996]
  We introduce Model Predictive Task Sampling (MPTS), a framework that bridges the task space and adaptation risk distributions.<n>MPTS employs a generative model to characterize the episodic optimization process and predicts task-specific adaptation risk via posterior inference.<n>MPTS seamlessly integrates into zero-shot, few-shot, and supervised finetuning settings.
  arXiv  Detail & Related papers  (2025-01-19T13:14:53Z)
• Self-Supervised Likelihood Estimation with Energy Guidance for Anomaly Segmentation in Urban Scenes [40.63116482332324]
  We design an energy-guided self-supervised framework for anomaly segmentation.<n>We exploit the strong context-dependent nature of the segmentation task.<n>Based on the proposed estimators, we devise an adaptive self-supervised training framework.
  arXiv  Detail & Related papers  (2023-02-14T03:54:32Z)
• Uncertainty-Aware Lidar Place Recognition in Novel Environments [11.30020653282995]
  We investigate the task of uncertainty-aware lidar place recognition. Each predicted place must have an associated uncertainty that can be used to identify and reject incorrect predictions. We introduce a novel evaluation protocol and present the first comprehensive benchmark for this task.
  arXiv  Detail & Related papers  (2022-10-04T04:06:44Z)
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks [44.61070965407907]
  Given the current state of the field, a standardized large-scale dataset of tasks across a range of modalities affected by distributional shifts is necessary. We propose the emphShifts dataset for evaluation of uncertainty estimates and robustness to distributional shift.
  arXiv  Detail & Related papers  (2021-07-15T16:59:34Z)
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving [77.39239190539871]
  We show how uncertainty estimation can be leveraged to enable safety critical image segmentation in autonomous driving. We introduce a new uncertainty measure based on disagreeing predictions as measured by a dissimilarity function. We show experimentally that our proposed approach is much less computationally intensive at inference time than competing methods.
  arXiv  Detail & Related papers  (2021-05-28T09:23:05Z)
• Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Learning [80.66108902283388]
  Multi-task learning methods should be used with caution for safety-critical applications, such as clinical risk prediction. Existing asymmetric multi-task learning methods tackle this negative transfer problem by performing knowledge transfer from tasks with low loss to tasks with high loss. We propose a novel temporal asymmetric multi-task learning model that performs knowledge transfer from certain tasks/timesteps to relevant uncertain tasks, based on feature-level uncertainty.
  arXiv  Detail & Related papers  (2020-06-23T06:01:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.