Related papers: VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

URL: http://arxiv.org/abs/2605.26144v1
Date: Fri, 22 May 2026 20:29:12 GMT
Title: VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents
Abstract summary: VISTA is a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents.<n>We define five prompt-information conditions that vary along two axes, visual/structural fidelity and stack constraint.<n>Each page in the benchmark is manually annotated with interactive UI components and around three visual anchor points.
Score: 25.141059096863255
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents. Unlike prior code generation benchmarks that focus on algorithmic tasks, VISTA targets realistic UI-centric development, where agents must produce functional, visually coherent applications from underspecified inputs. We define five prompt-information conditions that vary along two axes, visual/structural fidelity and stack constraint: (1) text only with free stack choice, (2) text with reference screenshots under three specified stacks, (3) text with reference screenshots under free stack choice, (4) text with screenshots and pruned Figma structure under a single specified stack, and (5) text with screenshots and pruned Figma structure under free stack choice. To enable robust evaluation, each page in the benchmark is manually annotated with interactive UI components and around three visual anchor points, addressing the well-known limitations of script-based testing tools such as Playwright in open-ended code generation settings. Evaluation combines DOM-grounded reference matching, behavior-specific browser tests, and CLIP-based visual similarity, jointly measuring structural alignment, behavioral completeness, and overall visual fidelity. We use VISTA to assess four agent systems drawn from two model families and two harnesses, finding that visual fidelity and functional correctness are partially decoupled across both input conditions and agents, and that agent editing style varies sharply but is largely orthogonal to task quality. VISTA establishes a rigorous and reproducible foundation for advancing agent-based software engineering research.

Related papers

UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation [35.72992007209751]
We introduce the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone.<n> UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications.<n>We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference.
arXiv Detail & Related papers (2026-07-07T14:08:55Z)
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence [31.954261925882452]
This survey examines systems that generate, edit, refine, or reason with code under visually grounded inputs and outputs.<n>We first formulate the field by the role that code plays in each task.<n>We then organize benchmarks and methods into four domains: Graphical User Interface, Scientific Visualization, Structured Graphics, and Frontier Tasks and Frameworks.
arXiv Detail & Related papers (2026-06-14T17:21:43Z)
Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation [19.889854990300595]
Iterative Retrieval-Augmented Generation (iRAG) has emerged as a powerful paradigm for answering complex multi-hop questions.<n>We present textbfChain of Evidence (CoE), a retriever-agnostic visual attribution framework.
arXiv Detail & Related papers (2026-05-02T06:40:42Z)
GALA: Multimodal Graph Alignment for Bug Localization in Automated Program Repair [40.228155198574356]
We propose GALA, a framework that shifts multimodal Automated Program Repair (APR) from implicit semantic guessing to explicit structural reasoning.<n>GALA operates in four stages: it first constructs an Image UI Graph to capture visual elements and their structural relationships; then performs file-level alignment by cross-referencing this UI graph with repository-level structures.<n>By enforcing both semantic and relational consistency across modalities, GALA establishes a highly accurate visual-to-code mapping.
arXiv Detail & Related papers (2026-04-09T11:06:25Z)
Multimodal OCR: Parse Anything from Documents [72.69545534962234]
dots.mocr treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets.<n>It reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction.<n>It supports end-to-end training over heterogeneous document elements.
arXiv Detail & Related papers (2026-03-13T14:42:21Z)
VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text? [51.02924254085878]
Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs.<n>We introduce VISTA-Bench, a benchmark from multimodal perception, reasoning, to unimodal understanding domains.
arXiv Detail & Related papers (2026-02-04T17:48:55Z)
Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization [50.13408999553116]
We propose RL-Text2Vis, the first reinforcement learning framework for Text2Vis generation.<n>Our method uses a novel multi-objective reward that jointly optimize textual accuracy, code validity, and visualization quality.<n>Our results establish GRPO as an effective strategy for structured, multimodal reasoning in visualization generation.
arXiv Detail & Related papers (2026-01-08T04:29:07Z)
AlignBench: Benchmarking Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs [27.133240420463807]
AlignBench is a benchmark that provides a new indicator of image-text alignment.<n>It evaluates detailed image-caption pairs generated by diverse image-to-text and text-to-image models.<n>Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators.
arXiv Detail & Related papers (2025-11-25T17:19:47Z)
Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
arXiv Detail & Related papers (2025-08-05T12:52:09Z)
OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition [79.852642726105]
We propose a unified paradigm for parsing visually-situated text across diverse scenarios. Specifically, we devise a universal model, called Omni, which can simultaneously handle three typical visually-situated text parsing tasks. In Omni, all tasks share the unified encoder-decoder architecture, the unified objective point-conditioned text generation, and the unified input representation.
arXiv Detail & Related papers (2024-03-28T03:51:14Z)
TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
arXiv Detail & Related papers (2023-06-06T03:37:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.