From Grounding to Planning: Benchmarking Bottlenecks in Web Agents

• URL: http://arxiv.org/abs/2409.01927v1
• Date: Tue, 3 Sep 2024 14:17:09 GMT
• Title: From Grounding to Planning: Benchmarking Bottlenecks in Web Agents
• Authors: Segev Shlomov, Ben wiesel, Aviad Sela, Ido Levy, Liane Galanti, Roy Abitbol,
• Abstract summary: General web-based agents are increasingly essential for interacting with complex web environments. Yet their performance in real-world web applications remains poor, yielding extremely low accuracy even with state-of-the-art frontier models. We sharpen the distinction between the planning and grounding components and conduct a novel analysis by refining experiments on the Mind2Web dataset.
• Score: 1.6135641587748402
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: General web-based agents are increasingly essential for interacting with complex web environments, yet their performance in real-world web applications remains poor, yielding extremely low accuracy even with state-of-the-art frontier models. We observe that these agents can be decomposed into two primary components: Planning and Grounding. Yet, most existing research treats these agents as black boxes, focusing on end-to-end evaluations which hinder meaningful improvements. We sharpen the distinction between the planning and grounding components and conduct a novel analysis by refining experiments on the Mind2Web dataset. Our work proposes a new benchmark for each of the components separately, identifying the bottlenecks and pain points that limit agent performance. Contrary to prevalent assumptions, our findings suggest that grounding is not a significant bottleneck and can be effectively addressed with current techniques. Instead, the primary challenge lies in the planning component, which is the main source of performance degradation. Through this analysis, we offer new insights and demonstrate practical suggestions for improving the capabilities of web agents, paving the way for more reliable agents.

Related papers

• How Important are Data Augmentations to Close the Domain Gap for Object Detection in Orbit? [15.550663626482903]
  We investigate the efficacy of data augmentations to close the domain gap in spaceborne computer vision. We propose two novel data augmentations specifically developed to emulate the visual effects observed in orbital imagery.
  arXiv  Detail & Related papers  (2024-10-21T08:24:46Z)
• ACTRESS: Active Retraining for Semi-supervised Visual Grounding [52.08834188447851]
  A previous study, RefTeacher, makes the first attempt to tackle this task by adopting the teacher-student framework to provide pseudo confidence supervision and attention-based supervision. This approach is incompatible with current state-of-the-art visual grounding models, which follow the Transformer-based pipeline. Our paper proposes the ACTive REtraining approach for Semi-Supervised Visual Grounding, abbreviated as ACTRESS.
  arXiv  Detail & Related papers  (2024-07-03T16:33:31Z)
• AI Agents That Matter [11.794931453828974]
  AI agents are an exciting new research direction, and agent development is driven by benchmarks. There is a narrow focus on accuracy without attention to other metrics. benchmarking needs of model and downstream developers have been conflated. Many agent benchmarks have inadequate holdout sets, and sometimes none at all.
  arXiv  Detail & Related papers  (2024-07-01T17:48:14Z)
• WebCanvas: Benchmarking Web Agents in Online Environments [29.278363444725628]
  WebCanvas is an innovative online evaluation framework for web agents. We open-source an agent framework with modules for reasoning, providing a foundation for the community to conduct online inference and evaluations. Our best-performing agent achieves a task success rate of 23.1% and a task completion rate of 48.8% on the Mind2Web-Live test set.
  arXiv  Detail & Related papers  (2024-06-18T07:58:33Z)
• SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving [27.731481134782577]
  Current models usually aggregate features from the neighboring frames to enhance the object representations for the task heads. These methods rely on the information from the future frames and suffer from high computational complexity. We introduce a stepwise spatial global-local aggregation network to solve these problems.
  arXiv  Detail & Related papers  (2024-05-29T08:12:51Z)
• Alice Benchmarks: Connecting Real World Re-Identification with the Synthetic [92.02220105679713]
  We introduce the Alice benchmarks, large-scale datasets providing benchmarks and evaluation protocols to the research community. Within the Alice benchmarks, two object re-ID tasks are offered: person and vehicle re-ID. As an important feature of our real target, the clusterability of its training set is not manually guaranteed to make it closer to a real domain adaptation test scenario.
  arXiv  Detail & Related papers  (2023-10-06T17:58:26Z)
• AI planning in the imagination: High-level planning on learned abstract search spaces [68.75684174531962]
  We propose a new method, called PiZero, that gives an agent the ability to plan in an abstract search space that the agent learns during training. We evaluate our method on multiple domains, including the traveling salesman problem, Sokoban, 2048, the facility location problem, and Pacman.
  arXiv  Detail & Related papers  (2023-08-16T22:47:16Z)
• Improving Long-Horizon Imitation Through Instruction Prediction [93.47416552953075]
  In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans.
  arXiv  Detail & Related papers  (2023-06-21T20:47:23Z)
• Goal-driven Self-Attentive Recurrent Networks for Trajectory Prediction [31.02081143697431]
  Human trajectory forecasting is a key component of autonomous vehicles, social-aware robots and video-surveillance applications. We propose a lightweight attention-based recurrent backbone that acts solely on past observed positions. We employ a common goal module, based on a U-Net architecture, which additionally extracts semantic information to predict scene-compliant destinations.
  arXiv  Detail & Related papers  (2022-04-25T11:12:37Z)
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [76.2621758731288]
  We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation. Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA) We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
  arXiv  Detail & Related papers  (2021-08-03T17:09:56Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.