NaviQAte: Functionality-Guided Web Application Navigation

• URL: http://arxiv.org/abs/2409.10741v1
• Date: Mon, 16 Sep 2024 21:18:39 GMT
• Title: NaviQAte: Functionality-Guided Web Application Navigation
• Authors: Mobina Shahbandeh, Parsa Alian, Noor Nashid, Ali Mesbah,
• Abstract summary: NaviQAte frames web application exploration as a question-and-answer task, generating action sequences for functionalities without requiring detailed parameters. Our three-phase approach utilizes advanced large language models like GPT-4o for complex decision-making and cost-effective models, such as GPT-4o mini, for simpler tasks.
• Score: 6.0759036120654315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end web testing is challenging due to the need to explore diverse web application functionalities. Current state-of-the-art methods, such as WebCanvas, are not designed for broad functionality exploration; they rely on specific, detailed task descriptions, limiting their adaptability in dynamic web environments. We introduce NaviQAte, which frames web application exploration as a question-and-answer task, generating action sequences for functionalities without requiring detailed parameters. Our three-phase approach utilizes advanced large language models like GPT-4o for complex decision-making and cost-effective models, such as GPT-4o mini, for simpler tasks. NaviQAte focuses on functionality-guided web application navigation, integrating multi-modal inputs such as text and images to enhance contextual understanding. Evaluations on the Mind2Web-Live and Mind2Web-Live-Abstracted datasets show that NaviQAte achieves a 44.23% success rate in user task navigation and a 38.46% success rate in functionality navigation, representing a 15% and 33% improvement over WebCanvas. These results underscore the effectiveness of our approach in advancing automated web application testing.

Related papers

• Infogent: An Agent-Based Framework for Web Information Aggregation [59.67710556177564]
  We introduce Infogent, a novel framework for web information aggregation. Experiments on different information access settings demonstrate Infogent beats an existing SOTA multi-agent search framework by 7%.
  arXiv  Detail & Related papers  (2024-10-24T18:01:28Z)
• Steward: Natural Language Web Automation [19.301371856154965]
  Large language models (LLMs) have demonstrated exceptional capabilities in serving as the foundation for AI assistants. We introduce Steward, a novel LLM-powered web automation tool designed to serve as a cost-effective, scalable, end-to-end solution for automating web interactions. We discuss various design and implementation challenges, including state representation, action sequence selection, system responsiveness, detecting task completion, and caching implementation.
  arXiv  Detail & Related papers  (2024-09-23T18:06:32Z)
• DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control [53.80518003412016]
  Building a general-purpose intelligent home-assistant agent skilled in diverse tasks by human commands is a long-term blueprint of embodied AI research. We study primitive mobile manipulations for embodied agents, i.e. how to navigate and interact based on an instructed verb-noun pair. We propose DISCO, which features non-trivial advancements in contextualized scene modeling and efficient controls.
  arXiv  Detail & Related papers  (2024-07-20T05:39:28Z)
• AutoWebGLM: A Large Language Model-based Web Navigating Agent [33.55199326570078]
  We develop the open AutoWebGLM based on ChatGLM3-6B. Inspired by human browsing patterns, we first design an HTML simplification algorithm to represent webpages. We then employ a hybrid human-AI method to build web browsing data for curriculum training.
  arXiv  Detail & Related papers  (2024-04-04T17:58:40Z)
• AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models [2.234037966956278]
  We introduce AllTogether, a standardized prompt template that enhances task context representation. We evaluate the efficacy of this approach through prompt learning and instruction finetuning based on open-source Llama-2 and API-accessible GPT models.
  arXiv  Detail & Related papers  (2023-10-20T11:10:14Z)
• Multimodal Web Navigation with Instruction-Finetuned Foundation Models [99.14209521903854]
  We study data-driven offline training for web agents with vision-language foundation models. We propose an instruction-following multimodal agent, WebGUM, that observes both webpage screenshots and HTML pages. We empirically demonstrate this recipe improves the agent's ability of grounded multimodal perception, HTML comprehension, and multi-step reasoning.
  arXiv  Detail & Related papers  (2023-05-19T17:44:34Z)
• ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments [56.194988818341976]
  Vision-language navigation is a task that requires an agent to follow instructions to navigate in environments. We propose ETPNav, which focuses on two critical skills: 1) the capability to abstract environments and generate long-range navigation plans, and 2) the ability of obstacle-avoiding control in continuous environments. ETPNav yields more than 10% and 20% improvements over prior state-of-the-art on R2R-CE and RxR-CE datasets.
  arXiv  Detail & Related papers  (2023-04-06T13:07:17Z)
• Towards Versatile Embodied Navigation [120.73460380993305]
  Vienna is a versatile embodied navigation agent that simultaneously learns to perform the four navigation tasks with one model. We empirically demonstrate that, compared with learning each visual navigation task individually, our agent achieves comparable or even better performance with reduced complexity.
  arXiv  Detail & Related papers  (2022-10-30T11:53:49Z)
• Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation [97.17517060585875]
  We present a unified approach to visual navigation using a novel modular transfer learning model. Our model can effectively leverage its experience from one source task and apply it to multiple target tasks. Our approach learns faster, generalizes better, and outperforms SoTA models by a significant margin.
  arXiv  Detail & Related papers  (2022-02-05T00:07:21Z)
• MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation [23.877609358505268]
  Recent work shows that map-like memory is useful for long-horizon navigation tasks. We propose the multiON task, which requires navigation to an episode-specific sequence of objects in a realistic environment. We examine how a variety of agent models perform across a spectrum of navigation task complexities.
  arXiv  Detail & Related papers  (2020-12-07T18:42:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.