ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding

• URL: http://arxiv.org/abs/2506.23491v3
• Date: Fri, 18 Jul 2025 06:03:26 GMT
• Title: ZonUI-3B: A Lightweight Vision-Language Model for Cross-Resolution GUI Grounding
• Abstract summary: ZonUI-3B is a lightweight Vision-Language Model (VLM) that can be fully trained on a single consumer-grade GPU.<n>It delivers performance comparable to significantly larger models on GUI grounding tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we present ZonUI-3B, a lightweight Vision-Language Model (VLM) that can be fully trained on a single consumer-grade GPU (RTX 4090) while delivering performance comparable to significantly larger models on GUI grounding tasks. The model incorporates several key innovations: (i) combine cross-platform, multi-resolution dataset of 24K examples from diverse sources including mobile, desktop, and web GUI screenshots to effectively address data scarcity in high-resolution desktop environments; (ii) a two-stage fine-tuning strategy, where initial cross-platform training establishes robust GUI understanding, followed by specialized fine-tuning on high-resolution data to significantly enhance model adaptability; and (iii) data curation and redundancy reduction strategies, demonstrating that randomly sampling a smaller subset with reduced redundancy achieves performance comparable to larger datasets, emphasizing data diversity over sheer volume. Empirical evaluation on standard GUI grounding benchmarks, including ScreenSpot, ScreenSpot-v2, and the challenging ScreenSpot-Pro, highlights ZonUI-3B's exceptional accuracy, achieving 84.9% on ScreenSpot and 86.4% on ScreenSpot-v2, surpassing prior models under 4B parameters. Ablation studies validate the critical role of balanced sampling and two-stage fine-tuning in enhancing robustness, particularly in high-resolution desktop scenarios. The ZonUI-3B is available at: https://github.com/Han1018/ZonUI-3B

Related papers

• Learnware for CSI Feedback: Scene-specific Small Models Can Do Big [118.52196588174786]
  Intelligent channel state information (CSI) feedback is essential for realizing the high capacity and spectral efficiency goals of future 6G systems.<n>Existing deep learning solutions face a trade-off between model generalization and scenario-specific performance.<n>We introduce a model repository-based deployment framework in which a central AI data center maintains a catalog of scene-specific CSI models.
  arXiv  Detail & Related papers  (2026-08-18T13:26:34Z)
• HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents [57.53434680014667]
  HyMobileAgent is a vision-native foundation model built on Hy3.0-VL-A3B.<n>We develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction.
  arXiv  Detail & Related papers  (2026-07-16T04:12:42Z)
• BAMI: Training-Free Bias Mitigation in GUI Grounding [72.1023567399117]
  We introduce textbfBias-Aware Manipulation Inference (BAMI), which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases.<n>BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting.
  arXiv  Detail & Related papers  (2026-05-07T17:59:31Z)
• POINTS-GUI-G: GUI-Grounding Journey [22.35782799756431]
  We introduce POINTS-GUIG-8B, which achieves state-of-the-art performance with scores of 59.9 on ScreenSpotPro, 66.0 on OSWorld-G, 95.7 on ScreenSpot-v2, and 49.9 on UIVision.<n>Our model's success is driven by three key factors: (1) Refined Data Engineering; (2) Improved Training Strategies; and (3) Reinforcement Learning with Verifiable Rewards.
  arXiv  Detail & Related papers  (2026-02-06T05:14:11Z)
• Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding [71.97466930670936]
  Grounding is a fundamental capability for building graphical user interface (GUI) agents.<n>In this paper, we investigate zoom as a strong yet underexplored prior to GUI grounding, and propose a training-free method, ZoomClick.<n> Experiments demonstrate that our method significantly boosts the performance of both general vision-language and specialized GUI grounding models.
  arXiv  Detail & Related papers  (2025-12-05T18:39:12Z)
• Yanyun-3: Enabling Cross-Platform Strategy Game Operation with Vision-Language Models [30.591909012704978]
  Yanyun-3 is a VLM-based agent that integrates Qwen2.5-VL for visual reasoning and UI-TARS for interface execution.<n>We propose a novel data organization principle to distinguish intra-sample fusion and inter-sample mixing of multimodal data.<n>Yanyun-3 offers a generalizable framework for GUI automation, with broader implications for robotics and autonomous systems.
  arXiv  Detail & Related papers  (2025-11-17T03:45:15Z)
• GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents [59.107657859025586]
  GUI-360$circ$ is a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs)<n>The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications.<n>The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space.
  arXiv  Detail & Related papers  (2025-11-06T12:19:02Z)
• Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation [65.3648667980258]
  Vision-language model (VLM) based GUI agents show promise for automating complex tasks, but face significant challenges in applying reinforcement learning (RL)<n>We propose DART, a Decoupled Agentic RL Training framework for GUI agents, which coordinates heterogeneous modules in a highly decoupled manner.<n>On the OSWorld benchmark, DART-GUI-7B achieves a 42.13% task success rate, a 14.61% absolute gain over the base model, and 7.34% higher than open-source SOTA.
  arXiv  Detail & Related papers  (2025-09-28T13:19:20Z)
• UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning [155.51875080423883]
  The development of autonomous agents for graphical user interfaces presents major challenges in artificial intelligence.<n>We present UI-TARS-2, a native GUI-centered agent model that addresses these challenges through a systematic training methodology.<n> Empirical evaluation demonstrates that UI-TARS-2 achieves significant improvements over its predecessor UI-TARS-1.5.
  arXiv  Detail & Related papers  (2025-09-02T17:44:45Z)
• UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding [14.42815380175443]
  We introduce UI-AGILE, a comprehensive framework enhancing GUI agents at both the training and inference stages.<n>For training, we propose a suite of improvements to the Supervised Fine-Tuning process.<n>For inference, we present Decomposed Grounding with Selection, a novel method that dramatically improves grounding accuracy on high-resolution displays.
  arXiv  Detail & Related papers  (2025-07-29T17:22:07Z)
• R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding [18.100091500983044]
  A critical challenge in GUI automation is the precise grounding of interface elements across diverse platforms.<n>Existing vision-only GUI agents directly ground elements from large and cluttered screenshots.<n>We introduce R-VLM, a novel GUI grounding approach that leverages zoomed-in region proposals for precise element localization.
  arXiv  Detail & Related papers  (2025-07-08T04:56:57Z)
• AugmentGest: Can Random Data Cropping Augmentation Boost Gesture Recognition Performance? [49.64902130083662]
  This paper proposes a comprehensive data augmentation framework that integrates geometric transformations, random variations, rotation, zooming and intensity-based transformations.<n>The proposed augmentation strategy is evaluated on three models: multi-stream e2eET, FPPR point cloud-based hand gesture recognition (HGR), and DD-Network.
  arXiv  Detail & Related papers  (2025-06-08T16:43:05Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)
• GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents [93.49577107524176]
  We propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding.<n>At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated ACTOR> token with all relevant visual patch tokens.<n>Experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks.
  arXiv  Detail & Related papers  (2025-06-03T17:59:08Z)
• Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning [28.291759852111586]
  We introduce a reinforcement learning (RL) based framework that incorporates three core strategies to improve GUI agent performance.<n>With only 3k training samples, our 7B- parameter model achieves state-of-the-art results among similarly sized models.<n> Notably, it attains 47.3% accuracy on the ScreenSpot-Pro dataset, outperforming much larger models, such as UI-TARS-72B, by a margin of 24.2%.
  arXiv  Detail & Related papers  (2025-05-18T11:22:04Z)
• GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents [16.72683291432717]
  name is the first reinforcement learning framework designed to enhance the capabilities of LVLMs in high-level real-world task scenarios.<n>name achieves superior performance using only 0.02% of the data compared to previous state-of-the-art methods like OS-Atlas.
  arXiv  Detail & Related papers  (2025-04-14T17:45:54Z)
• EfficientLLaVA:Generalizable Auto-Pruning for Large Vision-language Models [64.18350535770357]
  We propose an automatic pruning method for large vision-language models to enhance the efficiency of multimodal reasoning.<n>Our approach only leverages a small number of samples to search for the desired pruning policy.<n>We conduct extensive experiments on the ScienceQA, Vizwiz, MM-vet, and LLaVA-Bench datasets for the task of visual question answering.
  arXiv  Detail & Related papers  (2025-03-19T16:07:04Z)
• FLARES: Fast and Accurate LiDAR Multi-Range Semantic Segmentation [52.89847760590189]
  3D scene understanding is a critical yet challenging task in autonomous driving.<n>Recent methods leverage the range-view representation to improve processing efficiency.<n>We re-design the workflow for range-view-based LiDAR semantic segmentation.
  arXiv  Detail & Related papers  (2025-02-13T12:39:26Z)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
  This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
  arXiv  Detail & Related papers  (2025-01-21T17:48:10Z)
• LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving [52.83707400688378]
  LargeAD is a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets.<n>Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples.<n>Our approach delivers significant performance improvements over state-of-the-art methods in both linear probing and fine-tuning tasks for both LiDAR-based segmentation and object detection.
  arXiv  Detail & Related papers  (2025-01-07T18:59:59Z)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent [80.50062396585004]
  Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. We develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations. ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding.
  arXiv  Detail & Related papers  (2024-11-26T14:29:47Z)
• Contrastive Transformer Learning with Proximity Data Generation for Text-Based Person Search [60.626459715780605]
  Given a descriptive text query, text-based person search aims to retrieve the best-matched target person from an image gallery. Such a cross-modal retrieval task is quite challenging due to significant modality gap, fine-grained differences and insufficiency of annotated data. In this paper, we propose a simple yet effective dual Transformer model for text-based person search.
  arXiv  Detail & Related papers  (2023-11-15T16:26:49Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• Active Learning with Pseudo-Labels for Multi-View 3D Pose Estimation [18.768030475943213]
  We improve Active Learning for the problem of 3D pose estimation in a multi-view setting. We develop a framework that allows us to efficiently extend existing single-view AL strategies. We demonstrate additional performance gains by incorporating predicted pseudo-labels, which is a form of self-training.
  arXiv  Detail & Related papers  (2021-12-27T14:34:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.