Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

• URL: http://arxiv.org/abs/2609.15307v1
• Date: Mon, 14 Sep 2026 09:56:10 GMT
• Title: Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding
• Abstract summary: Confidence patterns in coordinate tokens are a better indicator than full-sequence confidence.<n>Negative samples offer more reliable learning signals than potentially noisy positive ones.<n>Our findings establish coordinate-token confidence as a powerful alternative to manual annotations for scalable GUI agent development.
• Score: 42.10524107620523
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Graphical User Interface (GUI) grounding is essential for autonomous agents to map natural language instructions to precise screen coordinates. However, existing supervised fine-tuning and reinforcement learning methods are constrained by the high cost of annotation, creating a scalability bottleneck. In this paper, we introduce a label-free test-time training paradigm driven by two key insights: (1) confidence patterns in coordinate tokens are a better indicator than full-sequence confidence, and (2) in sparse GUI coordinate spaces, negative samples offer more reliable learning signals than potentially noisy positive ones. We first propose Confidence-Anchored Learning (CAL), which utilizes coordinate-token confidence to filter pseudo-labels and assign distance-based binary rewards. Building on this, we develop Confidence-Anchored Negative Learning (CANL), which exclusively optimizes the model using negative samples to bypass the risks of incorrect positive samples. Experimental results demonstrate that CANL-7B achieves 92.1% on ScreenSpot-V2. On more challenging ScreenSpot-Pro, CANL-7B reaches 33.8%, an 8.9% absolute improvement over the base model. Our findings establish coordinate-token confidence as a powerful alternative to manual annotations for scalable GUI agent development.

Related papers

• Single-Channel EEG-Based Cognitive Load Assessment in Online Learning: A Hybrid Deep Learning Approach [0.0]
  We study whether a single-channel, consumer-grade EEG device can distinguish easy from difficult educational-video content.<n>We implement a hybrid CNN+LSTM+Attention model that combines the raw waveform with band-power features.
  arXiv  Detail & Related papers  (2026-07-02T07:13:01Z)
• BAMI: Training-Free Bias Mitigation in GUI Grounding [72.1023567399117]
  We introduce textbfBias-Aware Manipulation Inference (BAMI), which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases.<n>BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting.
  arXiv  Detail & Related papers  (2026-05-07T17:59:31Z)
• Linking Perception, Confidence and Accuracy in MLLMs [15.540652692247098]
  Confidence-Driven Reinforcement Learning (CDRL) uses original-noise image pairs to enhance sensitivity and robustly calibrate the model's confidence.<n> Confidence-Aware Test-Time Scaling (CA-TTS) dynamically coordinates Self-Consistency, Self-Reflection, and Visual Self-Check modules guided by confidence signals.<n>Our integrated framework establishes new state-of-the-art results with consistent 8.8% gains across four benchmarks.
  arXiv  Detail & Related papers  (2026-03-12T16:47:42Z)
• SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration [38.97678256807034]
  We introduce SafeGround, an uncertainty-aware framework for GUI grounding models.<n>It enables risk-aware predictions through calibrations before testing.<n>We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark.
  arXiv  Detail & Related papers  (2026-02-02T18:22:45Z)
• HyperClick: Advancing Reliable GUI Grounding via Uncertainty Calibration [18.315584342136997]
  HyperClick is a novel framework that enhances reliable GUI grounding through uncertainty calibration.<n>We show that HyperClick achieves state-of-the-art performance while providing well-calibrated confidence.
  arXiv  Detail & Related papers  (2025-10-31T08:07:02Z)
• HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions [69.14257241250046]
  We propose a new contrastive learning approach to train models for skeleton-based action recognition without labels. Our key contribution is a simple module, HaLP - to Hallucinate Latent Positives for contrastive learning. We show via experiments that using these generated positives within a standard contrastive learning framework leads to consistent improvements.
  arXiv  Detail & Related papers  (2023-04-01T21:09:43Z)
• To be Critical: Self-Calibrated Weakly Supervised Learning for Salient Object Detection [95.21700830273221]
  Weakly-supervised salient object detection (WSOD) aims to develop saliency models using image-level annotations. We propose a self-calibrated training strategy by explicitly establishing a mutual calibration loop between pseudo labels and network predictions. We prove that even a much smaller dataset with well-matched annotations can facilitate models to achieve better performance as well as generalizability.
  arXiv  Detail & Related papers  (2021-09-04T02:45:22Z)
• Improving Contrastive Learning by Visualizing Feature Transformation [37.548120912055595]
  In this paper, we attempt to devise a feature-level data manipulation, differing from data augmentation, to enhance the generic contrastive self-supervised learning. We first design a visualization scheme for pos/neg score (Pos/neg score indicates similarity of pos/neg pair.) distribution, which enables us to analyze, interpret and understand the learning process. Experiment results show that our proposed Feature Transformation can improve at least 6.0% accuracy on ImageNet-100 over MoCo baseline, and about 2.0% accuracy on ImageNet-1K over the MoCoV2 baseline.
  arXiv  Detail & Related papers  (2021-08-06T07:26:08Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Binary Classification from Positive Data with Skewed Confidence [85.18941440826309]
  Positive-confidence (Pconf) classification is a promising weakly-supervised learning method. In practice, the confidence may be skewed by bias arising in an annotation process. We introduce the parameterized model of the skewed confidence, and propose the method for selecting the hyper parameter.
  arXiv  Detail & Related papers  (2020-01-29T00:04:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.