Figma2Code: Automating Multimodal Design to Code in the Wild

• URL: http://arxiv.org/abs/2604.13648v1
• Date: Wed, 15 Apr 2026 09:17:09 GMT
• Title: Figma2Code: Automating Multimodal Design to Code in the Wild
• Abstract summary: We introduce Figma2Code, a new task that advances design-to-code into a multimodal setting.<n>We collect paired design images and their corresponding metadata files from the Figma community.<n>This process yields 3,055 samples, from which designers curate a balanced dataset of 213 high-quality cases.
• Score: 85.29510079067464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Front-end development constitutes a substantial portion of software engineering, yet converting design mockups into production-ready User Interface (UI) code remains tedious and costly. While recent work has explored automating this process with Multimodal Large Language Models (MLLMs), existing approaches typically rely solely on design images. As a result, they must infer complex UI details from images alone, often leading to degraded results. In real-world development workflows, however, design mockups are usually delivered as Figma files, a widely used tool for front-end design, that embed rich multimodal information (e.g., metadata and assets) essential for generating high-quality UI. To bridge this gap, we introduce Figma2Code, a new task that advances design-to-code into a multimodal setting and aims to automate design-to-code in the wild. Specifically, we collect paired design images and their corresponding metadata files from the Figma community. We then apply a series of processing operations, including rule-based filtering, human- and MLLM-based annotation and screening, and metadata refinement. This process yields 3,055 samples, from which designers curate a balanced dataset of 213 high-quality cases. Using this dataset, we benchmark ten state-of-the-art open-source and proprietary MLLMs. Our results show that while proprietary models achieve superior visual fidelity, they remain limited in layout responsiveness and code maintainability. Further experiments across modalities and ablation studies corroborate this limitation, partly due to models' tendency to directly map primitive visual attributes from Figma metadata.

Related papers

• UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling [57.126059929740926]
  We study Unified Visual Generation (UVG), where a single model produces diverse image-valued outputs through a unified multimodal interface.<n>We propose UniGen-AR, a framework that pairs a general-purpose multi-modal language model (MLLM) with an efficient visual auto-regressive decoder.<n>In our framework, the MLLM encodes free-form instructions and control signals into a unified sequence, which guides the VAR decoder to generate image-valued outputs for over 15 tasks spanning four families.
  arXiv  Detail & Related papers  (2026-07-27T08:35:09Z)
• OMTRA: A Multi-Task Generative Model for Structure-Based Drug Design [0.2578364360202627]
  We propose a unified approach in OMTRA, which flexibly performs many tasks relevant to drug design (SBDD)<n>We curate a dataset of 500M 3D molecular conformers, complementing protein-ligand data and expanding the chemical diversity available for training.<n>OMTRA obtains state of the art performance on pocket-conditioned de novo design and docking; however, the effects of large-scale pretraining and multi-task training are modest.
  arXiv  Detail & Related papers  (2025-12-04T18:46:35Z)
• VisCodex: Unified Multimodal Code Generation via Merging Vision and Coding Models [82.05514464090172]
  Multimodal large language models (MLLMs) have significantly advanced the integration of visual and textual understanding.<n>However, their ability to generate code from multimodal inputs remains limited.<n>We introduce VisCodex, a unified framework that seamlessly merges vision and coding language models.
  arXiv  Detail & Related papers  (2025-08-13T17:00:44Z)
• ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents [40.697759330690815]
  ScreenCoder is a modular multi-agent framework that decomposes the task into three interpretable stages: grounding, planning, and generation.<n>By assigning these distinct responsibilities to specialized agents, our framework achieves significantly higher robustness and fidelity than end-to-end approaches.<n>Our approach achieves state-of-the-art performance in layout accuracy, structural coherence, and code correctness.
  arXiv  Detail & Related papers  (2025-07-30T16:41:21Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts [53.568057283934714]
  We propose a Vision-Language Model (VLM)-based framework that generates content-aware text logo layouts.<n>We introduce two model techniques that reduce the computational cost for processing multiple glyph images simultaneously.<n>To support instruction tuning of our model, we construct two extensive text logo datasets that are five times larger than existing public datasets.
  arXiv  Detail & Related papers  (2024-11-18T10:04:10Z)
• PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM [58.67882997399021]
  Our research introduces a unified framework for automated graphic layout generation.<n>Our data-driven method employs structured text (JSON format) and visual instruction tuning to generate layouts.<n>We develop an automated text-to-poster system that generates editable posters based on users' design intentions.
  arXiv  Detail & Related papers  (2024-06-05T03:05:52Z)
• WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs [49.91550773480978]
  This paper introduces WebCode2M, a new dataset comprising 2.56 million instances, each containing a design image along with the corresponding webpage code and layout details.<n>To validate the effectiveness of WebCode2M, we introduce a baseline model based on the Vision Transformer (ViT), named WebCoder, and establish a benchmark for fair comparison.<n>The benchmarking results demonstrate that our dataset significantly improves the ability of MLLMs to generate code from webpage designs.
  arXiv  Detail & Related papers  (2024-04-09T15:05:48Z)
• LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer [80.61492265221817]
  Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs is skill-demanding, time-consuming, and non-scalable to batch production. Generative models emerge to make design automation scalable but it remains non-trivial to produce designs that comply with designers' desires.
  arXiv  Detail & Related papers  (2022-12-19T21:57:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.