EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants

• URL: http://arxiv.org/abs/2608.29387v1
• Date: Sat, 29 Aug 2026 17:54:02 GMT
• Title: EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants
• Abstract summary: Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve.<n>We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios.<n>We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs.
• Score: 10.510136510866099
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models can generate interactive web interfaces, but reliable generative UI requires maintaining an executable artifact as user requests evolve. We introduce EvoGenUI-Bench, a benchmark for multi-turn interface maintenance comprising 150 five-turn tasks and 750 turns across three scenarios: information presentation, executable interaction, and tool-grounded external state. We execute generated artifacts in a browser and evaluate them using screenshots, source and DOM evidence, actor traces, and runtime logs. Beyond turn-level and episode-level success, we measure cross-turn retention with Adjacent Pass Retention. Across eight models, even the strongest achieves 74.9% Turn Pass while completing only 37.3% of five-turn episodes; APR further falls to 52.4% on tool-grounded tasks. Diagnostic analysis shows that presentation failures center on information architecture, interaction failures on derived-state propagation and affordance binding, and tool-grounded failures additionally involve external-state grounding and requirement decomposition. These results reframe generative UI evaluation from judging isolated outputs to testing whether interface behavior, derived state, external state, and assistant claims remain synchronized as the artifact evolves.

Related papers

• EvolveScaler: Synthesizing Information-Evolution Contexts via Executable State Machines and Natural-Language Rendering [57.87517047585447]
  We introduce EvolveScaler, a code-driven framework that defines information evolution before rendering it as natural language.<n>We instantiate EvolveScaler with 117 task prototypes and 159 final-question operators across five difficulty levels spanning approximately 7 to 1,200 events per instance.<n>On the very_long tier, the strongest model reaches 59.3% avg@5, while six models score below 10%.
  arXiv  Detail & Related papers  (2026-09-08T08:40:05Z)
• UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations [68.29650478987561]
  UI-Mate is a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning.<n>It sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld and 66.2% on WindowsAgentArena.
  arXiv  Detail & Related papers  (2026-08-16T20:59:59Z)
• Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? [3.9561795512466413]
  Current benchmarks primarily measure end-task success or single-frame grounding.<n>This is difficult because inference, remote input, app rendering, and screenshot capture are asynchronous.<n>We introduce Desktop-Delta Bench (DDB), an offline step-level benchmark with 2,013 human-verified instances from novel, multi-app Linux trajectories.
  arXiv  Detail & Related papers  (2026-07-28T17:49:51Z)
• AgentEscapeBench: Evaluating Out-of-Domain Tool-Grounded Reasoning in LLM Agents [35.967227261886435]
  AgentEscapeBench tests whether agents can infer, execute, and revise novel tool-use procedures under explicit long-range dependency constraints.<n>Experiments with sixteen LLM agents and human participants show that performance drops sharply as dependency depth increases.<n>Trajectory analysis attributes model failures mainly to breakdowns in long-range state tracking, clue adherence, and intermediate-result propagation.
  arXiv  Detail & Related papers  (2026-05-08T15:59:27Z)
• Practical Limits of Autonomous Test Repair: A Multi-Agent Case Study with LLM-Driven Discovery and Self-Correction [0.0]
  We present an industrial case study of a multi-agent autonomous testing system evaluated using anonymized execution data from a production-like enterprise UI testing prototype.<n>The system evolves from human-directed testing toward High-autonomy feature discovery and test execution.<n>Our findings show that unrestricted autonomy leads to unstable and often misleading outcomes, while constrained autonomy transforms such systems into operationally viable.
  arXiv  Detail & Related papers  (2026-05-02T14:39:55Z)
• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation [98.38575149237442]
  VLAA-GUI is a modular GUI framework built around three integrated components.<n>A mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step.<n>A mandatory Loop Breaker provides multi-tier switching interaction mode after repeated failures.
  arXiv  Detail & Related papers  (2026-04-23T07:42:37Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning [155.51875080423883]
  The development of autonomous agents for graphical user interfaces presents major challenges in artificial intelligence.<n>We present UI-TARS-2, a native GUI-centered agent model that addresses these challenges through a systematic training methodology.<n> Empirical evaluation demonstrates that UI-TARS-2 achieves significant improvements over its predecessor UI-TARS-1.5.
  arXiv  Detail & Related papers  (2025-09-02T17:44:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.