Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development

• URL: http://arxiv.org/abs/2609.02088v1
• Date: Wed, 02 Sep 2026 04:27:45 GMT
• Title: Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development
• Abstract summary: We present RILA, an execution-driven agent that puts browser rendering in the loop, iteratively editing generated code from runtime interaction feedback.<n>On IWR-Bench, RILA consistently improves both interaction and visual fidelity across foundation models.
• Score: 11.314424904357212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models have achieved remarkable progress in front-end web development, generating interactive webpages from multimodal references such as screenshots and interaction videos. However, existing work largely emphasizes visual metrics such as aesthetics and layout similarity, while overlooking the more critical validation of interactive functionality. We present RILA, an execution-driven agent that puts browser rendering in the loop, iteratively editing generated code from runtime interaction feedback. RILA introduces an Action Interaction Verification (AIV) module that replays the reference interaction trajectory on the generated webpage to collect grounded execution-aware observations, and an Execution-aware Rendering Score (ERS) that jointly measures interaction correctness and visual fidelity to guide iterative optimization. We further build an execution-verified data synthesis pipeline that produces diverse, high-quality training data, offering gains complementary to inference-time optimization. On IWR-Bench, RILA consistently improves both interaction and visual fidelity across foundation models. Notably, with our training pipeline, RILA lifts the compact Qwen3.5-9B backbone from 40.40% to 57.52%, surpassing far larger one-shot generators, including the 1T-parameter Kimi-K2.6 (55.61%) and the proprietary GPT-5.5 (55.74%).

Related papers

• UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp [14.339105360523831]
  Real-world multimodal browsing involves three distinct information-flow patterns: text-only, image-to-text, and text-to-image.<n>We introduce UNIBROWSE, a unified data pipeline that simultaneously generates training data covering all three patterns.<n>The resulting UNIBROWSE agent achieves state-of-the-art performance on multimodal BrowseComp benchmarks.
  arXiv  Detail & Related papers  (2026-07-12T04:13:27Z)
• UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation [35.72992007209751]
  We introduce the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone.<n> UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications.<n>We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference.
  arXiv  Detail & Related papers  (2026-07-07T14:08:55Z)
• Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages [22.804937497354754]
  We introduce WebIGBench, the first benchmark to evaluate code generation for interactive webpages with complex interactions.<n>By combining manually designed interaction paths with UI automation, we collected 103 complex webpages from real-world websites.<n>This benchmark covers 5 popular interactive action types (e.g., click, input) involving 871 distinct interactive actions.
  arXiv  Detail & Related papers  (2026-05-29T05:45:50Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• Covering Human Action Space for Computer Use: Data Synthesis and Benchmark [59.01879944842542]
  Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude.<n>Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust.<n>We propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions.
  arXiv  Detail & Related papers  (2026-05-12T17:59:58Z)
• 1D-Bench: A Benchmark for Iterative UI Code Generation with Visual Feedback in Real-World [5.904589000032003]
  We introduce 1D-Bench, a benchmark grounded in real e-commerce, where each instance provides a reference rendering and an exported intermediate representation.<n>1D is short for one day, representing the efficient completion of design-to-code tasks in less than one day.
  arXiv  Detail & Related papers  (2026-02-20T17:46:51Z)
• FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback [92.67587639164908]
  We present FronTalk, a benchmark for front-end code generation with multi-modal feedback.<n>We focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues.<n> Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature.
  arXiv  Detail & Related papers  (2025-12-05T23:28:09Z)
• MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling [115.74855199827596]
  MiroThinker is an open-source research agent designed to advance tool-augmented reasoning and information-seeking capabilities.<n>Unlike previous agents that only scale up model size or context length, MiroThinker explores interaction scaling at the model level.
  arXiv  Detail & Related papers  (2025-11-14T18:52:07Z)
• WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning [51.14454312533818]
  WebGen-Agent is a novel website-generation agent that leverages comprehensive and multi-level visual feedback.<n>We introduce textitStep-GRPO with Screenshot and GUI-agent Feedback to improve the ability of LLMs to act as the reasoning engine of WebGen-Agent.<n>WebGen-Agent increases the accuracy of Claude-3.5-Sonnet from 26.4% to 51.9% and its appearance score from 3.0 to 3.9, outperforming the previous state-of-the-art agent system.
  arXiv  Detail & Related papers  (2025-09-26T17:59:51Z)
• InterFormer: Real-time Interactive Image Segmentation [80.45763765116175]
  Interactive image segmentation enables annotators to efficiently perform pixel-level annotation for segmentation tasks. The existing interactive segmentation pipeline suffers from inefficient computations of interactive models. We propose a method named InterFormer that follows a new pipeline to address these issues.
  arXiv  Detail & Related papers  (2023-04-06T08:57:00Z)
• Asynchronous Interaction Aggregation for Action Detection [43.34864954534389]
  We propose the Asynchronous Interaction Aggregation network (AIA) that leverages different interactions to boost action detection. There are two key designs in it: one is the Interaction Aggregation structure (IA) adopting a uniform paradigm to model and integrate multiple types of interaction; the other is the Asynchronous Memory Update algorithm (AMU) that enables us to achieve better performance.
  arXiv  Detail & Related papers  (2020-04-16T07:03:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.