Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements

• URL: http://arxiv.org/abs/2603.14448v1
• Date: Sun, 15 Mar 2026 15:47:47 GMT
• Title: Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements
• Abstract summary: Multimodal Large Language Model (MLLM)-based Graphical User Interface (GUI) agents develop rapidly.<n>Existing GUI agents typically fine-tune MLLM on massive datasets to handle challenges in understanding instructions and UI interfaces.<n>We propose ZoomUI that leverages inference scaling to guide common MLLMs in progressively anchor instruction elements to increasingly detailed interface elements.
• Score: 40.21437107734778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Model (MLLM)-based Graphical User Interface (GUI) agents develop rapidly, with visual grounding that maps natural language instructions to target UI elements serving as the core capability. Existing GUI agents typically fine-tune MLLM on massive datasets to handle challenges in understanding instructions and UI interfaces, which not only incurs high data annotation costs but also makes performance dependent on data quality and distribution. To avoid such cumbersome yet ineffective training, we notice that complex UI interfaces can be decomposed into basic visual elements directly understandable by common MLLMs. Consequently, we propose ZoomUI that leverages inference scaling to guide common MLLMs in progressively anchor instruction elements to increasingly detailed interface elements. Specifically, ZoomUI first optimizes the latent thinking to transform original instruction into element visual features description, and subsequently leverages internal attention to iteratively zoom in target element interface region. Evaluations on extensive benchmarks demonstrate that ZoomUI reaches or even surpasses SOTA baselines.

Related papers

• What's Missing in Screen-to-Action? Towards a UI-in-the-Loop Paradigm for Multimodal GUI Reasoning [58.734995044357845]
  We propose an innovative GUI reasoning paradigm called UI-in-the-Loop (UILoop)<n>Our approach treats the GUI reasoning task as a cyclic Screen-UI elements-Action process.<n>By enabling Multimodal Large Language Models (MLLMs) to explicitly learn the localization, semantic functions, and practical usage of key UI elements, UILoop achieves precise discovery and performs interpretable reasoning.
  arXiv  Detail & Related papers  (2026-04-08T12:12:09Z)
• AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement [44.11867590785016]
  We propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding.<n>Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions.<n>In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements.
  arXiv  Detail & Related papers  (2026-03-18T07:26:18Z)
• Structuring GUI Elements through Vision Language Models: Towards Action Space Generation [43.932266242034025]
  Multimodal large language models (MLLMs) have emerged as pivotal tools in enhancing human-computer interaction.<n>This paper focuses on the application of MLLMs in the field of graphical user interface (GUI) elements structuring.<n>We introduce an IoU-Augmented Maximum Likelihood (IAML) training paradigm to bolster visual module capabilities.
  arXiv  Detail & Related papers  (2025-08-22T10:14:15Z)
• DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning [53.42606072841585]
  We introduce DiMo-GUI, a training-free framework for GUI grounding.<n>Instead of treating the GUI as a monolithic image, our method splits the input into textual elements and iconic elements.<n>When predictions are ambiguous or incorrect, DiMo-GUI dynamically focuses attention by generating candidate focal regions.
  arXiv  Detail & Related papers  (2025-06-12T03:13:21Z)
• UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis [15.429065788185522]
  Vision-based Graphical User Interface (GUI) agents utilize human-like vision perception capabilities to enhance productivity on digital devices.<n>In this paper, we explore unexplored challenges in this task including element-to-screen ratio, unbalanced element type, and implicit instruction.<n>We propose a new GUI instruction grounding benchmark UI-I2E-Bench, which is designed to address the limitations of existing benchmarks.
  arXiv  Detail & Related papers  (2025-04-15T14:56:21Z)
• TRISHUL: Towards Region Identification and Screen Hierarchy Understanding for Large VLM based GUI Agents [0.6827423171182154]
  TRISHUL is a training-free framework that enhances generalist LVLMs for holistic GUI comprehension.<n>Our results demonstrate TRISHUL's superior performance in action grounding across the ScreenSpot, VisualWebBench, AITW, and Mind2Web datasets.<n>For GUI referring, TRISHUL surpasses the ToL agent on the ScreenPR benchmark, setting a new standard for robust and adaptable GUI comprehension.
  arXiv  Detail & Related papers  (2025-02-12T09:12:30Z)
• AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs [54.58905728115257]
  We propose the textbfAutoGUI pipeline for automatically annotating UI elements with detailed functionality descriptions at scale.<n>We construct a high-quality AutoGUI-704k dataset using the proposed pipeline, featuring diverse and detailed functionality annotations.<n>Our dataset remarkably enhances VLM's UI grounding capabilities and exhibits significant scaling effects.
  arXiv  Detail & Related papers  (2025-02-04T03:39:59Z)
• Falcon-UI: Understanding GUI Before Following User Instructions [57.67308498231232]
  We introduce an instruction-free GUI navigation dataset, termed Insight-UI dataset, to enhance model comprehension of GUI environments.<n> Insight-UI dataset is automatically generated from the Common Crawl corpus, simulating various platforms.<n>We develop the GUI agent model Falcon-UI, which is initially pretrained on Insight-UI dataset and subsequently fine-tuned on Android and Web GUI datasets.
  arXiv  Detail & Related papers  (2024-12-12T15:29:36Z)
• GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding [73.9254861755974]
  This paper introduces a new dataset, termed GUI-World, which features meticulously crafted Human-MLLM annotations.<n>We evaluate the capabilities of current state-of-the-art MLLMs, including Image LLMs and Video LLMs, in understanding various types of GUI content.
  arXiv  Detail & Related papers  (2024-06-16T06:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.