GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation

• URL: http://arxiv.org/abs/2603.14724v1
• Date: Mon, 16 Mar 2026 01:57:56 GMT
• Title: GameUIAgent: An LLM-Powered Framework for Automated Game UI Design with Structured Intermediate Representation
• Abstract summary: Game UI design requires consistent visual assets across rarity tiers yet remains a predominantly manual process.<n>We present GameUIAgent LLM-powered agentic framework that translates natural language descriptions into editable Figma designs.<n>A six-stage neuro-symbolic pipeline combines LLM generation, deterministic post-processing, and a Vision-Language Model (VLM)-guided Reflection Controller (RC) for iterative self-correction.
• Score: 10.74393288520677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Game UI design requires consistent visual assets across rarity tiers yet remains a predominantly manual process. We present GameUIAgent, an LLM-powered agentic framework that translates natural language descriptions into editable Figma designs via a Design Spec JSON intermediate representation. A six-stage neuro-symbolic pipeline combines LLM generation, deterministic post-processing, and a Vision-Language Model (VLM)-guided Reflection Controller (RC) for iterative self-correction with guaranteed non-regressive quality. Evaluated across 110 test cases, three LLMs, and three UI templates, cross-model analysis establishes a game-domain failure taxonomy (rarity-dependent degradation; visual emptiness) and uncovers two key empirical findings. A Quality Ceiling Effect (Pearson r=-0.96, p<0.01) suggests that RC improvement is bounded by headroom below a quality threshold -- a visual-domain counterpart to test-time compute scaling laws. A Rendering-Evaluation Fidelity Principle reveals that partial rendering enhancements paradoxically degrade VLM evaluation by amplifying structural defects. Together, these results establish foundational principles for LLM-driven visual generation agents in game production.

Related papers

• UniGen-AR: Unifying Visual Generation with Auto-Regressive Modeling [57.126059929740926]
  We study Unified Visual Generation (UVG), where a single model produces diverse image-valued outputs through a unified multimodal interface.<n>We propose UniGen-AR, a framework that pairs a general-purpose multi-modal language model (MLLM) with an efficient visual auto-regressive decoder.<n>In our framework, the MLLM encodes free-form instructions and control signals into a unified sequence, which guides the VAR decoder to generate image-valued outputs for over 15 tasks spanning four families.
  arXiv  Detail & Related papers  (2026-07-27T08:35:09Z)
• Employing Vision-Language Models for Face Image Quality Assessment [0.023141219541346198]
  Face Image Quality Assessment (FIQA) is a crucial control step in biometric pipelines.<n>State-of-the-art FIQA methods achieve high utility but typically operate as "black boxes"<n>We investigate the potential of off-the-shelf Vision-Language Models (VLMs) to bridge this gap by performing FIQA in a zero-shot setting.
  arXiv  Detail & Related papers  (2026-05-17T14:57:52Z)
• On Integrating Resilience and Human Oversight into LLM-Assisted Modeling Workflows for Digital Twins [0.0]
  We present three critical design principles for integrating resilience and oversight into Digital Twins.<n>A key contribution is detailed characterization of LLM-induced errors across model descriptions of varying detail and complexity.
  arXiv  Detail & Related papers  (2026-03-26T20:38:03Z)
• ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval [64.14282916266998]
  Composed Image Retrieval aims to retrieve target images based on a hybrid query comprising a reference image and a modification text.<n>We propose ReCALL, a model-agnostic framework that follows a diagnose-generate-refine pipeline.<n>Experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-02T04:52:54Z)
• LP-LLM: End-to-End Real-World Degraded License Plate Text Recognition via Large Multimodal Models [4.497411606350301]
  Real-world License Plate Recognition (LPR) faces significant challenges from severe degradations such as motion blur, low resolution, and complex illumination.<n>The prevailing "restoration-then-recognition" two-stage paradigm suffers from a fundamental flaw: the pixel-level optimization objectives of image restoration models are misaligned with the semantic goals of character recognition.<n>We propose an end-to-end structure-aware multimodal reasoning framework based on Qwen3-VL.
  arXiv  Detail & Related papers  (2026-01-14T03:32:55Z)
• Investigate the Low-level Visual Perception in Vision-Language based Image Quality Assessment [7.969076042774561]
  We introduce a low-level distortion perception task that requires models to classify specific distortion types.<n>Our analysis shows that although MLLMs are structurally capable of representing such distortions, they tend to overfit training templates.<n>We show that improving the alignment of the vision encoder dramatically enhances distortion recognition accuracy, increasing it from 14.92% to 84.43%.
  arXiv  Detail & Related papers  (2025-12-10T12:06:47Z)
• LLM as a Neural Architect: Controlled Generation of Image Captioning Models Under Strict API Contracts [48.83701310501069]
  We present NN-Caption, an LLM-guided neural architecture search pipeline.<n>It generates runnable image-captioning models by composing CNN encoders from LEMUR's classification backbones.<n>This work presents a pipeline that integrates prompt-based code generation with automatic evaluation.
  arXiv  Detail & Related papers  (2025-12-07T10:47:28Z)
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback [33.127607245587576]
  We introduce a framework that enables MLLMs to learn complex visual reasoning from only raw images.<n>We demonstrate that this relative ease provides an ideal reward signal for optimization via Reinforcement Learning.<n>The RRVF-trained model not only outperforms existing MLLMs and supervised fine-tuning baselines but also exhibits superior generalization.
  arXiv  Detail & Related papers  (2025-07-28T12:21:19Z)
• Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought [58.321044666612174]
  Vad-R1 is an end-to-end MLLM-based framework for Video Anomaly Reasoning.<n>We design a Perception-to-Cognition Chain-of-Thought (P2C-CoT) that simulates the human process of recognizing anomalies.<n>We also propose an improved reinforcement learning algorithm AVA-GRPO, which explicitly incentivizes the anomaly reasoning capability of MLLMs.
  arXiv  Detail & Related papers  (2025-05-26T12:05:16Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next-token prediction is the fundamental principle for training large language models (LLMs)<n>We introduce R1-SGG, a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset.<n>We design a set of graph-centric rewards, including three recall-based variants -- Hard Recall, Hard Recall+Relax, and Soft Recall.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models [73.40350756742231]
  Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored.
  arXiv  Detail & Related papers  (2024-02-12T18:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.