MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs

• URL: http://arxiv.org/abs/2412.15310v1
• Date: Thu, 19 Dec 2024 15:02:33 GMT
• Title: MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs
• Authors: Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, Michael R. Lyu,
• Abstract summary: Multi-Page Resource-Aware Webpage (MRWeb) generation task transforms UI designs into multi-page, functional web UIs with internal/external navigation, image loading, and backend routing. Our study applies existing methods to the MRWeb problem using a newly curated dataset of 500 websites (300 synthetic, 200 real-world). Specifically, we identify the best metric to evaluate the similarity of the web UI, assess the impact of the resource list on MRWeb generation, analyze MLLM limitations, and evaluate the effectiveness of the MRWeb tool in real-world.
• Score: 50.274447094978996
• License:
• Abstract: Multi-page websites dominate modern web development. However, existing design-to-code methods rely on simplified assumptions, limiting to single-page, self-contained webpages without external resource connection. To address this gap, we introduce the Multi-Page Resource-Aware Webpage (MRWeb) generation task, which transforms UI designs into multi-page, functional web UIs with internal/external navigation, image loading, and backend routing. We propose a novel resource list data structure to track resources, links, and design components. Our study applies existing methods to the MRWeb problem using a newly curated dataset of 500 websites (300 synthetic, 200 real-world). Specifically, we identify the best metric to evaluate the similarity of the web UI, assess the impact of the resource list on MRWeb generation, analyze MLLM limitations, and evaluate the effectiveness of the MRWeb tool in real-world workflows. The results show that resource lists boost navigation functionality from 0% to 66%-80% while facilitating visual similarity. Our proposed metrics and evaluation framework provide new insights into MLLM performance on MRWeb tasks. We release the MRWeb tool, dataset, and evaluation framework to promote further research.

Related papers

• Multi-Record Web Page Information Extraction From News Websites [83.88591755871734]
  In this paper, we focus on the problem of extracting information from web pages containing many records. To address this gap, we created a large-scale, open-access dataset specifically designed for list pages. Our dataset contains 13,120 web pages with news lists, significantly exceeding existing datasets in both scale and complexity.
  arXiv  Detail & Related papers  (2025-02-20T15:05:00Z)
• WebRPG: Automatic Web Rendering Parameters Generation for Visual Presentation [24.99791278208309]
  We introduce Web Rendering Parameters Generation (WebRPG), a new task that aims at automating the generation for visual presentation of web pages based on their HTML code. We present baseline models, utilizing VAE to manage numerous elements and rendering parameters, along with custom HTML embedding for capturing essential semantic and hierarchical information from HTML.
  arXiv  Detail & Related papers  (2024-07-22T09:35:43Z)
• Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs [112.89665642941814]
  Multimodal large language models (MLLMs) have shown impressive success across modalities such as image, video, and audio. Current MLLMs are surprisingly poor at understanding webpage screenshots and generating their corresponding HTML code. We propose a benchmark consisting of a new large-scale webpage-to-code dataset for instruction tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:59:46Z)
• AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation [54.17246674188208]
  Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website. We introduce the paradigm of generating web scrapers with large language models (LLMs) and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently.
  arXiv  Detail & Related papers  (2024-04-19T09:59:44Z)
• VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding? [115.60866817774641]
  Multimodal Large Language models (MLLMs) have shown promise in web-related tasks. evaluating their performance in the web domain remains a challenge due to the lack of comprehensive benchmarks. bench is a multimodal benchmark designed to assess the capabilities of MLLMs across a variety of web tasks.
  arXiv  Detail & Related papers  (2024-04-09T02:29:39Z)
• A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding [66.6468787004067]
  We introduce the Wikipedia Webpage suite (WikiWeb2M) containing 2M pages with all of the associated image, text, and structure data. We design a novel attention mechanism Prefix Global, which selects the most relevant image and text content as global tokens to attend to the rest of the webpage for context.
  arXiv  Detail & Related papers  (2023-05-05T16:38:05Z)
• Layout-aware Webpage Quality Assessment [31.537331183733837]
  We propose a novel layout-aware webpage quality assessment model currently deployed in our search engine. We employ the meta-data that describes a webpage, i.e., Document Object Model (DOM) tree, as the input of our model. To assess webpage quality from complex DOM tree data, we propose a graph neural network (GNN) based method.
  arXiv  Detail & Related papers  (2023-01-28T10:27:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.