UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding

• URL: http://arxiv.org/abs/2604.14113v1
• Date: Wed, 15 Apr 2026 17:32:28 GMT
• Title: UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding
• Abstract summary: Test-time zoom-in methods apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case.<n>We propose textbfUI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty problem.
• Score: 55.22866422166757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GUI grounding, which localizes interface elements from screenshots given natural language queries, remains challenging for small icons and dense layouts. Test-time zoom-in methods improve localization by cropping and re-running inference at higher resolution, but apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case. We propose \textbf{UI-Zoomer}, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty quantification problem. A confidence-aware gate fuses spatial consensus among stochastic candidates with token-level generation confidence to selectively trigger zoom-in only when localization is uncertain. When triggered, an uncertainty-driven crop sizing module decomposes prediction variance into inter-sample positional spread and intra-sample box extent, deriving a per-instance crop radius via the law of total variance. Extensive experiments on ScreenSpot-Pro, UI-Vision, and ScreenSpot-v2 demonstrate consistent improvements over strong baselines across multiple model architectures, achieving gains of up to +13.4\%, +10.3\%, and +4.2\% respectively, with no additional training required.

Related papers

• SA-GEM: Scale-Adaptive and Geospatial Evidence-Modulated Token Pruning for Efficient Remote Sensing Large Vision-Language Models [52.99440326540357]
  RS-LVLMs have advanced multimodal understanding of Earth observation imagery, yet their performance is constrained by high-resolution processing.<n>Existing token pruning methods rely on scale-agnostic resolution policies and isolated importance cues.<n>We present Scale-Adaptive and Geospatial Evidence-Modulated Token Pruning (SA-GEM), a plug-and-play framework that unifies task-adaptive token granularity allocation with holistic geospatial token importance modulation.
  arXiv  Detail & Related papers  (2026-08-15T06:50:11Z)
• FoundationGeo: Learning Spatial Pixel-Wise Fields for Monocular Metric Geometry [52.232826445242644]
  FoundationGeo bridges relative and metric prediction via spatial calibration and principled data design.<n>Stage 1 learns a high-fidelity, affine-invariant geometry model by initializing with DINOv3.<n>Stage 2 moves beyond global scaling by introducing pixel-wise calibration fields for metric estimation.
  arXiv  Detail & Related papers  (2026-07-13T14:10:01Z)
• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding [56.38477453373368]
  LazyMCoT is a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty.<n>LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.
  arXiv  Detail & Related papers  (2026-06-15T03:17:44Z)
• AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding [8.941979904606308]
  Vision-Language Models (VLMs) have enabled autonomous GUI agents that translate natural language instructions into executable screen coordinates.<n>Existing zoom-in strategies rely on fixed anchors, grids, or reinforcement learning.<n>We propose AutoFocus, a training-free, uncertainty-aware active visual search framework for GUI grounding.
  arXiv  Detail & Related papers  (2026-05-04T14:18:46Z)
• SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration [38.97678256807034]
  We introduce SafeGround, an uncertainty-aware framework for GUI grounding models.<n>It enables risk-aware predictions through calibrations before testing.<n>We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark.
  arXiv  Detail & Related papers  (2026-02-02T18:22:45Z)
• Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding [71.97466930670936]
  Grounding is a fundamental capability for building graphical user interface (GUI) agents.<n>In this paper, we investigate zoom as a strong yet underexplored prior to GUI grounding, and propose a training-free method, ZoomClick.<n> Experiments demonstrate that our method significantly boosts the performance of both general vision-language and specialized GUI grounding models.
  arXiv  Detail & Related papers  (2025-12-05T18:39:12Z)
• Improving GUI Grounding with Explicit Position-to-Coordinate Mapping [40.201918480639954]
  Current approaches generate coordinates as text tokens directly from visual features, forcing the model to infer complex position-to-pixel mappings implicitly.<n>We propose RULER tokens as explicit coordinate markers, letting the model reference positions similar to gridlines on a map and adjust rather than generate coordinates from scratch.<n>Experiments on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent gains in grounding accuracy, with the largest improvements on high-resolution interfaces.
  arXiv  Detail & Related papers  (2025-10-03T17:59:34Z)
• RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization [50.75654397516163]
  We propose RelayFormer, a unified framework that adapts to varying resolutions and modalities.<n> RelayFormer partitions inputs into fixed-size sub-images and introduces Global-Local Relay (GLR) tokens.<n>This enables efficient exchange of global cues, such as semantic or temporal consistency, while preserving fine-grained manipulation artifacts.
  arXiv  Detail & Related papers  (2025-08-13T03:35:28Z)
• GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding [51.497245303008015]
  Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction.<n>Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards.<n>We show that GUI-G$2$, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro.
  arXiv  Detail & Related papers  (2025-07-21T17:53:42Z)
• Scale-adaptive UAV Geo-localization via Height-aware Partition Learning [22.72128308133626]
  UAV Geo-Localization faces significant challenges due to the drastic appearance discrepancy between dronecaptured images and satellite views.<n>Existing methods typically assume a consistent scaling factor across views and rely on predefined partition alignment.<n>We propose a scale-adaptive partition learning framework that leverages known drone flight height to predict scale factors and dynamically adjust feature extraction.
  arXiv  Detail & Related papers  (2024-12-16T08:13:05Z)
• HeightFormer: A Multilevel Interaction and Image-adaptive Classification-regression Network for Monocular Height Estimation with Aerial Images [10.716933766055755]
  This paper presents a comprehensive solution for monocular height estimation in remote sensing. It features the Multilevel Interaction Backbone (MIB) and Image-adaptive Classification-regression Height Generator (ICG) The ICG dynamically generates height partition for each image and reframes the traditional regression task.
  arXiv  Detail & Related papers  (2023-10-12T02:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.