FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows

• URL: http://arxiv.org/abs/2608.21460v1
• Date: Thu, 20 Aug 2026 16:59:33 GMT
• Title: FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows
• Abstract summary: A major contributor to this performance gap is the lack of high quality human workflow data.<n>FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories.<n>We show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models.
• Score: 2.535605790826847
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as \textsc{Claude-Opus-5} and \textsc{GPT-5.6-Sol} on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing \textsc{Qwen3.8-27B} outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.

Related papers

• Figma2Code: Automating Multimodal Design to Code in the Wild [85.29510079067464]
  We introduce Figma2Code, a new task that advances design-to-code into a multimodal setting.<n>We collect paired design images and their corresponding metadata files from the Figma community.<n>This process yields 3,055 samples, from which designers curate a balanced dataset of 213 high-quality cases.
  arXiv  Detail & Related papers  (2026-04-15T09:17:09Z)
• DesignSense: A Human Preference Dataset and Reward Modeling Framework for Graphic Layout Generation [23.876231579797803]
  We introduce DesignSense-10k, a large-scale dataset of 10,235 human-annotated preference pairs for graphic layout evaluation.<n>We use semantic grouping, layout prediction, filtering, clustering, and VLM-based refinement to produce high-quality comparison pairs.<n>Our analysis shows that frontier VLMs remain unreliable overall and fail catastrophically on the full four-class task.
  arXiv  Detail & Related papers  (2026-02-26T19:03:05Z)
• Learning Design-Score Manifold to Guide Diffusion Models for Offline Optimization [15.663508678977468]
  This paper introduces ManGO, a diffusion-based framework that learns the design-score manifold.<n>ManGO unifies the design-score interdependencies holistically, attaining generalization beyond training data.<n>It outperforms 24 single- and 10 multi-objective optimization methods across diverse domains.
  arXiv  Detail & Related papers  (2025-06-06T02:11:10Z)
• Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model [55.25659103706409]
  This framework achieves state-of-the-art performance for our designed foundation model, YingLong.<n>YingLong is a non-causal, bidirectional attention encoder-only transformer trained through masked token recovery.<n>We release four foundation models ranging from 6M to 300M parameters, demonstrating superior results in zero-shot tasks.
  arXiv  Detail & Related papers  (2025-05-20T14:31:06Z)
• SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation [81.36747103102459]
  Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications.<n>Current state-of-the-art methods focus on training innovative architectural designs on confined datasets.<n>We investigate the impact of scaling up EHPS towards a family of generalist foundation models.
  arXiv  Detail & Related papers  (2025-01-16T18:59:46Z)
• FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks [8.659674736978555]
  We introduce Foveation-based Explanations (FovEx), a novel XAI method inspired by human vision.<n>Our method achieves state-of-the-art performance on both transformers and convolutional models.
  arXiv  Detail & Related papers  (2024-08-04T19:37:30Z)
• Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient [52.2669490431145]
  PropEn is inspired by'matching', which enables implicit guidance without training a discriminator. We show that training with a matched dataset approximates the gradient of the property of interest while remaining within the data distribution.
  arXiv  Detail & Related papers  (2024-05-28T11:30:19Z)
• Compositional Generative Inverse Design [69.22782875567547]
  Inverse design, where we seek to design input variables in order to optimize an underlying objective function, is an important problem. We show that by instead optimizing over the learned energy function captured by the diffusion model, we can avoid such adversarial examples. In an N-body interaction task and a challenging 2D multi-airfoil design task, we demonstrate that by composing the learned diffusion model at test time, our method allows us to design initial states and boundary shapes.
  arXiv  Detail & Related papers  (2024-01-24T01:33:39Z)
• Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning [19.28860833813788]
  Existing models commonly train a visual encoder with weak cross-modal supervision signals. We propose a novel textbfVisually-textbfAsymmetric cotextbfNsistentextbfCy textbfLearning (textscVancl) approach to capture fine-grained visual and layout features.
  arXiv  Detail & Related papers  (2023-10-23T10:37:22Z)
• Challenging the Myth of Graph Collaborative Filtering: a Reasoned and Reproducibility-driven Analysis [50.972595036856035]
  We present a code that successfully replicates results from six popular and recent graph recommendation models. We compare these graph models with traditional collaborative filtering models that historically performed well in offline evaluations. By investigating the information flow from users' neighborhoods, we aim to identify which models are influenced by intrinsic features in the dataset structure.
  arXiv  Detail & Related papers  (2023-08-01T09:31:44Z)
• Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow [22.540665278228975]
  We propose VAIDA, a novel benchmark creation paradigm for NLP. VAIDA focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies. We find that VAIDA decreases effort, frustration, mental, and temporal demands of crowdworkers and analysts.
  arXiv  Detail & Related papers  (2023-02-09T04:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.