InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?

• URL: http://arxiv.org/abs/2604.27419v1
• Date: Thu, 30 Apr 2026 04:49:34 GMT
• Title: InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
• Abstract summary: Real-world development is constrained by a critical bottleneck: the semantic misalignment between ambiguous, low-quality instructions from non-expert users and model understanding.<n>We introduce InteractWeb-Bench, the first multimodal interactive benchmark for website generation under non-expert low-code user conditions.
• Score: 20.180246350877045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the advancement of multimodal large language models (MLLMs) and coding agents, the website development has shifted from manual programming to agent-based project-level code synthesis. Existing benchmarks rely on idealized assumptions, especially for well-structured, information-rich inputs and static execution settings. In contrast, real-world development is constrained by a critical bottleneck: the semantic misalignment between ambiguous, low-quality instructions from non-expert users and model understanding, which results in a failure mode that we term blind execution. To address this gap, we introduce InteractWeb-Bench, the first multimodal interactive benchmark for website generation under non-expert low-code user conditions. InteractWeb-Bench introduces four types of user agents and persona-driven instruction perturbations to systematically simulate diverse user behaviors, including ambiguity, redundancy, and contradiction, grounded in requirement engineering defect taxonomies. We develop an interactive execution environment for agents, featuring a unified action space comprising Clarify, Implement, Verify, and Submit, enabling iterative intent refinement, code synthesis, and visual feedback-based validation. Extensive experiments and analysis reveal that frontier MLLM-based agents remain trapped in blind execution, exposing limitations in intent recognition and adaptive interaction.

Related papers

• Framework and Benchmark for Code-Driven Agentic Testing in Web Development [25.642063881090337]
  We introduce textbfCode-driven textbfAgentic textbfTesting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs.<n>We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs.
  arXiv  Detail & Related papers  (2026-08-31T10:07:51Z)
• Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages [22.804937497354754]
  We introduce WebIGBench, the first benchmark to evaluate code generation for interactive webpages with complex interactions.<n>By combining manually designed interaction paths with UI automation, we collected 103 complex webpages from real-world websites.<n>This benchmark covers 5 popular interactive action types (e.g., click, input) involving 871 distinct interactive actions.
  arXiv  Detail & Related papers  (2026-05-29T05:45:50Z)
• WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models [40.87133775066985]
  We introduce Web, a multimodal benchmark that provides unified lifecycle evaluation of web engineering capability.<n>Web spans three input modalities (text, image) and three task types (generation, editing, repair)<n>For evaluation, we adopt a checklist-guided LLM-as-a-Judge protocol for editing and repair, and propose a novel Agent-as-a-Judge paradigm for generation that autonomously generated websites in a real browser.
  arXiv  Detail & Related papers  (2026-04-20T13:09:38Z)
• FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback [92.67587639164908]
  We present FronTalk, a benchmark for front-end code generation with multi-modal feedback.<n>We focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues.<n> Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature.
  arXiv  Detail & Related papers  (2025-12-05T23:28:09Z)
• WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality [62.43165871914528]
  We introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development.<n>WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics.<n>In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias.
  arXiv  Detail & Related papers  (2025-10-21T12:16:04Z)
• Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting [92.57796055887995]
  We introduce ECHO, a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents.<n> ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts.<n>We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation.
  arXiv  Detail & Related papers  (2025-10-11T18:11:09Z)
• Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping [57.024913536420264]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task.<n>We present the first systematic investigation of MLLMs in generating interactive webpages.
  arXiv  Detail & Related papers  (2024-11-05T17:40:03Z)
• Compromising Embodied Agents with Contextual Backdoor Attacks [69.71630408822767]
  Large language models (LLMs) have transformed the development of embodied intelligence. This paper uncovers a significant backdoor security threat within this process. By poisoning just a few contextual demonstrations, attackers can covertly compromise the contextual environment of a black-box LLM.
  arXiv  Detail & Related papers  (2024-08-06T01:20:12Z)
• On the Multi-turn Instruction Following for Conversational Web Agents [83.51251174629084]
  We introduce a new task of Conversational Web Navigation, which necessitates sophisticated interactions that span multiple turns with both the users and the environment. We propose a novel framework, named self-reflective memory-augmented planning (Self-MAP), which employs memory utilization and self-reflection techniques.
  arXiv  Detail & Related papers  (2024-02-23T02:18:12Z)
• Signifiers as a First-class Abstraction in Hypermedia Multi-Agent Systems [0.6595290783361959]
  We build on concepts and methods from Affordance Theory and Human-Computer Interaction to introduce signifiers as a first-class abstraction in Web-based Multi-Agent Systems. We define a formal model for the contextual exposure of signifiers in hypermedia environments that aims to drive affordance exploitation.
  arXiv  Detail & Related papers  (2023-02-14T10:54:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.