BAMI: Training-Free Bias Mitigation in GUI Grounding

• URL: http://arxiv.org/abs/2605.06664v1
• Date: Thu, 07 May 2026 17:59:31 GMT
• Title: BAMI: Training-Free Bias Mitigation in GUI Grounding
• Abstract summary: We introduce textbfBias-Aware Manipulation Inference (BAMI), which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases.<n>BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting.
• Score: 72.1023567399117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: GUI grounding is a critical capability for enabling GUI agents to execute tasks such as clicking and dragging. However, in complex scenarios like the ScreenSpot-Pro benchmark, existing models often suffer from suboptimal performance. Utilizing the proposed \textbf{Masked Prediction Distribution (MPD)} attribution method, we identify that the primary sources of errors are twofold: high image resolution (leading to precision bias) and intricate interface elements (resulting in ambiguity bias). To address these challenges, we introduce \textbf{Bias-Aware Manipulation Inference (BAMI)}, which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases. Our extensive experimental results demonstrate that BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting. For instance, applying our method to the TianXi-Action-7B model boosts its accuracy on the ScreenSpot-Pro benchmark from 51.9\% to 57.8\%. Furthermore, ablation studies confirm the robustness of the BAMI approach across diverse parameter configurations, highlighting its stability and effectiveness. Code is available at https://github.com/Neur-IO/BAMI.

Related papers

• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding [56.38477453373368]
  LazyMCoT is a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty.<n>LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.
  arXiv  Detail & Related papers  (2026-06-15T03:17:44Z)
• SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibration [38.97678256807034]
  We introduce SafeGround, an uncertainty-aware framework for GUI grounding models.<n>It enables risk-aware predictions through calibrations before testing.<n>We apply SafeGround on multiple GUI grounding models for the challenging ScreenSpot-Pro benchmark.
  arXiv  Detail & Related papers  (2026-02-02T18:22:45Z)
• UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning [51.54456545661045]
  We introduce the Instruction-as-Reasoning paradigm, treating instructions as dynamic analytical pathways that offer distinct perspectives.<n>To achieve this, we propose a two-stage training framework: supervised fine-tuning and reinforcement learning.<n>Our resulting models, UI-Ins-7B and UI-Ins-32B, achieve state-of-the-art results on five challenging grounding benchmarks.
  arXiv  Detail & Related papers  (2025-10-23T07:18:32Z)
• UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding [16.939058522414836]
  We introduce UI-AGILE for enhancing GUI agents at both training and inference.<n>For training, we propose a suite of improvements to the Supervised Fine-Tuning (SFT) process.<n>For inference, we present decomposed grounding with selection to dramatically improve grounding accuracy on high-resolution displays.
  arXiv  Detail & Related papers  (2025-07-29T17:22:07Z)
• GTA1: GUI Test-time Scaling Agent [97.58177633084915]
  Graphical user interface (GUI) agents autonomously complete tasks across platforms (eg, Linux) by sequentially decomposing user instructions into action proposals.<n>This paper investigates the aforementioned challenges with our textbfGUI textbfTest-time Scaling textbfAgent, namely GTA1.
  arXiv  Detail & Related papers  (2025-07-08T08:52:18Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models [152.29364079385635]
  As pre-trained models grow bigger, the fine-tuning process can be time-consuming and computationally expensive. We propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning and (ii) resource-efficient inference.
  arXiv  Detail & Related papers  (2021-10-30T03:29:47Z)
• Mutual-Information Based Few-Shot Classification [34.95314059362982]
  We introduce Transductive Infomation Maximization (TIM) for few-shot learning. Our method maximizes the mutual information between the query features and their label predictions for a given few-shot task. We propose a new alternating-direction solver, which speeds up transductive inference over gradient-based optimization.
  arXiv  Detail & Related papers  (2021-06-23T09:17:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.