V2P: From Background Suppression to Center Peaking for Robust GUI Grounding Task

• URL: http://arxiv.org/abs/2508.13634v1
• Date: Tue, 19 Aug 2025 08:47:44 GMT
• Title: V2P: From Background Suppression to Center Peaking for Robust GUI Grounding Task
• Abstract summary: Valley-to-Peak method is inspired by how humans visually process and interact with GUI elements.<n>Model trained by V2P achieves 92.3% and 50.5% on two benchmarks ScreenSpot-v2 and ScreenSpot-Pro.
• Score: 16.500878734275936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Precise localization of GUI elements is crucial for the development of GUI agents. Traditional methods rely on bounding box or center-point regression, neglecting spatial interaction uncertainty and visual-semantic hierarchies. Recent methods incorporate attention mechanisms but still face two key issues: (1) ignoring processing background regions causes attention drift from the desired area, and (2) uniform labeling fails to distinguish between center and edges of the target UI element, leading to click imprecision. Inspired by how humans visually process and interact with GUI elements, we propose the Valley-to-Peak (V2P) method to address these issues. To mitigate background distractions, V2P introduces a suppression attention mechanism that minimizes the model's focus on irrelevant regions to highlight the intended region. For the issue of center-edge distinction, V2P applies a Fitts' Law-inspired approach by modeling GUI interactions as 2D Gaussian heatmaps where the weight gradually decreases from the center towards the edges. The weight distribution follows a Gaussian function, with the variance determined by the target's size. Consequently, V2P effectively isolates the target area and teaches the model to concentrate on the most essential point of the UI element. The model trained by V2P achieves the performance with 92.3% and 50.5% on two benchmarks ScreenSpot-v2 and ScreenSpot-Pro. Ablations further confirm each component's contribution, highlighting V2P's generalizability for precise GUI grounding tasks.

Related papers

• AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding [8.941979904606308]
  Vision-Language Models (VLMs) have enabled autonomous GUI agents that translate natural language instructions into executable screen coordinates.<n>Existing zoom-in strategies rely on fixed anchors, grids, or reinforcement learning.<n>We propose AutoFocus, a training-free, uncertainty-aware active visual search framework for GUI grounding.
  arXiv  Detail & Related papers  (2026-05-04T14:18:46Z)
• UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding [55.22866422166757]
  Test-time zoom-in methods apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case.<n>We propose textbfUI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty problem.
  arXiv  Detail & Related papers  (2026-04-15T17:32:28Z)
• V2P: Visual Attention Calibration for GUI Grounding via Background Suppression and Center Peaking [33.84965941459024]
  Valley-to-Peak method is inspired by how humans visually process and interact with GUI elements.<n>Model trained by V2P achieves 92.4% and 52.5% on two benchmarks.
  arXiv  Detail & Related papers  (2026-01-11T12:54:54Z)
• Improving GUI Grounding with Explicit Position-to-Coordinate Mapping [40.201918480639954]
  Current approaches generate coordinates as text tokens directly from visual features, forcing the model to infer complex position-to-pixel mappings implicitly.<n>We propose RULER tokens as explicit coordinate markers, letting the model reference positions similar to gridlines on a map and adjust rather than generate coordinates from scratch.<n>Experiments on ScreenSpot, ScreenSpot-V2, and ScreenSpot-Pro show consistent gains in grounding accuracy, with the largest improvements on high-resolution interfaces.
  arXiv  Detail & Related papers  (2025-10-03T17:59:34Z)
• Point2RBox-v3: Self-Bootstrapping from Point Annotations via Integrated Pseudo-Label Refinement and Utilization [58.42853147118086]
  Point2RBox-v3 is first model to employ dynamic pseudo labels for label assignment.<n>Our solution gives competitive performance, especially in scenarios with large variations in object size or sparse object occurrences.
  arXiv  Detail & Related papers  (2025-09-30T14:01:59Z)
• GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding [51.497245303008015]
  Graphical User Interface (GUI) grounding maps natural language instructions to precise interface locations for autonomous interaction.<n>Motivated by human clicking behavior that naturally forms Gaussian distributions centered on target elements, we introduce GUI Gaussian Grounding Rewards.<n>We show that GUI-G$2$, substantially outperforms state-of-the-art method UI-TARS-72B, with the most significant improvement of 24.7% on ScreenSpot-Pro.
  arXiv  Detail & Related papers  (2025-07-21T17:53:42Z)
• Visual Test-time Scaling for GUI Agent Grounding [61.609126885427386]
  We introduce RegionFocus, a visual test-time scaling approach for Vision Language Model Agents.<n>Our approach dynamically zooms in on relevant regions, reducing background clutter and improving grounding accuracy.<n>We observe significant performance gains of 28+% on Screenspot-pro and 24+% on WebVoyager benchmarks.
  arXiv  Detail & Related papers  (2025-05-01T17:45:59Z)
• Point2RBox-v2: Rethinking Point-supervised Oriented Object Detection with Spatial Layout Among Instances [50.80161958767447]
  We present Point2RBox-v2, an approach to explore the spatial layout among instances for learning point-supervised OOD.<n>Our solution is elegant and lightweight, yet it is expected to give a competitive performance especially in densely packed scenes.
  arXiv  Detail & Related papers  (2025-02-06T18:07:25Z)
• RD-VIO: Robust Visual-Inertial Odometry for Mobile Augmented Reality in Dynamic Environments [55.864869961717424]
  It is typically challenging for visual or visual-inertial odometry systems to handle the problems of dynamic scenes and pure rotation. We design a novel visual-inertial odometry (VIO) system called RD-VIO to handle both of these problems.
  arXiv  Detail & Related papers  (2023-10-23T16:30:39Z)
• Polarized Self-Attention: Towards High-quality Pixel-wise Regression [19.2303932008785]
  This paper presents the Polarized Self-Attention(PSA) block that incorporates two critical designs towards high-quality pixel-wise regression. Experimental results show that PSA boosts standard baselines by $2-4$ points, and boosts state-of-the-arts by $1-2$ points on 2D pose estimation and semantic segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-07-02T01:03:11Z)
• Pixel-in-Pixel Net: Towards Efficient Facial Landmark Detection in the Wild [104.61677518999976]
  We propose Pixel-in-Pixel Net (PIPNet) for facial landmark detection. The proposed model is equipped with a novel detection head based on heatmap regression. To further improve the cross-domain generalization capability of PIPNet, we propose self-training with curriculum.
  arXiv  Detail & Related papers  (2020-03-08T12:23:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.