Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

• URL: http://arxiv.org/abs/2606.13192v1
• Date: Thu, 11 Jun 2026 11:00:16 GMT
• Title: Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
• Abstract summary: We propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning.<n>UX-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963.
• Score: 14.935343848369486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs' ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.

Related papers

• LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents [52.61417375636918]
  We present LLaDA-UI, a 16.7B- parameter MoE-based, block-wise diffusion vision-language GUI agent.<n>We show that LLaDA-UI substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported GUI benchmarks.
  arXiv  Detail & Related papers  (2026-09-09T05:53:39Z)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning [58.734995044357845]
  We propose an innovative GUI reasoning paradigm called UI-in-the-Loop (UILoop)<n>Our approach treats the GUI reasoning task as a cyclic Screen-UI elements-Action process.<n>By enabling Multimodal Large Language Models (MLLMs) to explicitly learn the localization, semantic functions, and practical usage of key UI elements, UILoop achieves precise discovery and performs interpretable reasoning.
  arXiv  Detail & Related papers  (2026-04-08T12:12:09Z)
• GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents [19.27396264271709]
  GUI-CEval is the first comprehensive benchmark for Chinese mobile GUI agents built entirely on physical device environments.<n>It spans 201 mainstream apps across four device types and adopts a two-level structure that evaluates both atomic abilities and realistic application-level performance along five dimensions: perception, planning, reflection, execution, and evaluation.
  arXiv  Detail & Related papers  (2026-03-16T09:45:33Z)
• FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoning [7.78727102442322]
  We propose the FC-MIR framework: leveraging sampling and adaptive concatenation, it cuts visual redundancy to boost inference efficiency.<n>We further expand task scope to explore generating post-prediction operations and search suggestions, and introduce a fine-grained metric to evaluate the practical utility of summaries, predictions, and suggestions.<n>We deploy the framework in a real-world setting, integrating UI perception and UI-Agent proxies to lay a foundation for future progress in this field.
  arXiv  Detail & Related papers  (2025-12-22T07:21:07Z)
• UI-UG: A Unified MLLM for UI Understanding and Generation [19.7078650905834]
  We introduce UI-UG (a unified MLLM for UI Understanding and Generation), integrating both capabilities.<n>For understanding tasks, we employ Supervised Fine-tuning (SFT) combined with Group Relative Policy Optimization (GRPO) to enhance fine-grained understanding.<n>For generation tasks, we further use Direct Preference Optimization (DPO) to make our model generate human-preferred UIs.
  arXiv  Detail & Related papers  (2025-09-29T06:59:09Z)
• ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation [51.297873393639456]
  ArtifactsBench is a framework for automated visual code generation evaluation.<n>Our framework renders each generated artifact and captures its dynamic behavior through temporal screenshots.<n>We construct a new benchmark of 1,825 diverse tasks and evaluate over 30 leading Large Language Models.
  arXiv  Detail & Related papers  (2025-07-07T12:53:00Z)
• Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding [45.81445929920235]
  We introduce WiserUI-Bench, a novel benchmark for assessing models' multimodal understanding of UI/UX design.<n>It includes 300 diverse real-world UI image pairs, each consisting of two design variants A/B-tested at scale by actual companies.<n>Our benchmark supports two core tasks: (1) selecting the more effective UI/UX design by predicting the A/B test verified winner and (2) assessing how well a model, given the winner, can explain its effectiveness in alignment with expert reasoning.
  arXiv  Detail & Related papers  (2025-05-08T08:00:32Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
  This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
  arXiv  Detail & Related papers  (2025-01-21T17:48:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.