LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection

• URL: http://arxiv.org/abs/2608.09723v1
• Date: Mon, 10 Aug 2026 15:23:52 GMT
• Title: LookAgain: Closed-Loop GUI Grounding with Visually Grounded Reflection
• Abstract summary: Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks.<n>But their performance degrades sharply on small targets, densely packed controls and out-of-distribution interfaces.<n>We propose LookAgain, a closed-loop GUI grounder driven by post-prediction visual reflection.
• Score: 42.672264792698904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks, yet their performance degrades sharply on small targets, densely packed controls and out-of-distribution interfaces. We attribute this gap to a paradigmatic limitation shared by existing approaches: none of them treats a produced coordinate as a hypothesis to be reflected upon and revised under new visual evidence. This manifests as three coupled issues: 1) Lack of post-hoc reflection. The prediction is frozen at the moment of emission, leaving no internal mechanism to challenge or refine it. 2) Visual evidence decoupled from the prediction. The auxiliary visual evidence is gathered to support the upcoming coordinate rather than to scrutinise the one already committed to. 3) Refinement over views, not over predictions. The iterative zoom-in refines the inspected region instead of inheriting a previous coordinate as a spatial prior to be corrected. In this paper, we propose LookAgain, a closed-loop GUI grounder driven by post-prediction visual reflection. LookAgain reformulates grounding as a multi-turn predict-look-again-refine process with two primitives: "locate" posts a coordinate hypothesis, renders a marker on the image and appends a local patch of the predicted region. It anchors the next reasoning step to the previous prediction as a spatial prior; "confirm" accepts or reject the hypothesis and terminates the procedure. We train the LookAgain grounder with SFT on constructed reflective trajectories as a cold start, followed by GRPO with terminal grounding correctness as the sole reward. Extensive experiments show that LookAgain consistently improves performance on both refusal-aware and general GUI grounding benchmarks, achieving state-of-the-art results. Comprehensive ablations further verify the effectiveness of the proposed framework.

Related papers

• MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents [74.83535434786145]
  MNIST-PRO is a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints.<n>We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas.
  arXiv  Detail & Related papers  (2026-08-31T16:05:39Z)
• Glance, Scrutinize, and Think: Advancing Video Anomaly Detection from Training-Free to Agentic Reasoning [27.676141768874732]
  Video Anomaly Detection (VAD) aims to identify anomalous events and localize their temporal intervals.<n>Existing approaches exhibit a "when-what" dissociation: traditional methods localize when anomalies occur but lack semantic understanding.<n>We propose Glance then Scrutinize (GtS), a training-free framework using static and dynamic textual guidance.<n>Experiments show that GtS substantially surpasses training-free baselines, while the agentic model delivers both higher accuracy and faster inference.
  arXiv  Detail & Related papers  (2026-08-07T17:07:49Z)
• CAVE: Competence-Aware Visual Boundary Evidence Alignment for Video Temporal Grounding [37.61837019441122]
  Large vision-language models (LVLMs) have achieved substantial performance gains in Video Temporal Grounding (VTG) through reinforcement learning (RL)<n>Existing methods rely on outcome correctness rewards that evaluate only the final predicted intervals, leaving boundary-related visual evidence insufficiently constrained.<n>We propose Competence-Aware Visual Boundary Alignment (CAVE), which augments localization optimization with boundary-specific visual evidence rewards to timestamp misalignment.
  arXiv  Detail & Related papers  (2026-08-03T11:27:29Z)
• Reasoning as a Double-Edged Sword: Architecture and Cross-Stage Robustness in Vision-Language-Action Models [27.2709460237741]
  Policy that reasons before acting should absorb a perturbed input better than one that maps observations directly to actions.<n>We test this premise head-on across three models that span the reasoning spectrum.<n>We find that the latent-iterative model is by far the least robust.
  arXiv  Detail & Related papers  (2026-07-20T10:20:32Z)
• SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs [74.89208610190595]
  We present SPOT-E, a test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning.<n> SPOT-E yields consistent gains and improved robustness under visual corruptions.
  arXiv  Detail & Related papers  (2026-06-18T13:56:30Z)
• Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models [29.66841995436342]
  Modern vision-language models (VLMs) remain unreliable on metric distance queries.<n>We introduce textbfViewDiag, a controlled multi-view evaluation protocol built from Hypersim, ScanNet, and KITTI360.<n>We show that stable predictions may reflect prior-driven collapse rather than evidence-sensitive reasoning.
  arXiv  Detail & Related papers  (2026-06-01T18:06:08Z)
• Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models [77.3748853516374]
  We propose a training-free, model-intrinsic grounding method for vision-language models.<n>We compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map.<n>We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule.
  arXiv  Detail & Related papers  (2026-04-09T16:51:42Z)
• Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? [93.10694819127608]
  We propose a unified evaluation pipeline for forecasting methods with real-world perception inputs. Our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data.
  arXiv  Detail & Related papers  (2023-06-15T17:03:14Z)
• Patch-level Gaze Distribution Prediction for Gaze Following [49.93340533068501]
  We introduce the patch distribution prediction ( PDP) method for gaze following training. We show that our model regularizes the MSE loss by predicting better heatmap distributions on images with larger annotation variances. Experiments show that our model bridging the gap between the target prediction and in/out prediction subtasks, showing a significant improvement on both subtasks on public gaze following datasets.
  arXiv  Detail & Related papers  (2022-11-20T19:25:15Z)
• Which to Match? Selecting Consistent GT-Proposal Assignment for Pedestrian Detection [23.92066492219922]
  The fixed Intersection over Union (IoU) based assignment-regression manner still limits their performance. We introduce one geometric sensitive search algorithm as a new assignment and regression metric. Specifically, we boost the MR-FPPI under R$_75$ by 8.8% on Citypersons dataset.
  arXiv  Detail & Related papers  (2021-03-18T08:54:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.