iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception

• URL: http://arxiv.org/abs/2512.22009v1
• Date: Fri, 26 Dec 2025 12:09:15 GMT
• Title: iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception
• Authors: Sarthak Mehrotra, Sairam V C Rebbapragada, Mani Hemanth Reddy Bonthu, Vineeth N Balasubramanian,
• Abstract summary: We introduce iSHIFT: Implicit Slow-fast Hybrid Inference with Flexible Tokens, a lightweight agent that integrates latent thinking with a perception control module.<n>iSHIFT enables an MLLM to switch between a slow mode, which leverages detailed visual grounding for high precision and a fast mode that uses global cues for efficiency.<n>Despite its compact 2.5B size, iSHIFT matches state-of-the-art performance on multiple benchmark datasets.
• Score: 27.22349186465607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal Large Language Models (MLLMs) show strong potential for interpreting and interacting with complex, pixel-rich Graphical User Interface (GUI) environments. However, building agents that are both efficient for high-level tasks and precise for fine-grained interactions remains challenging. GUI agents must perform routine actions efficiently while also handling tasks that demand exact visual grounding, yet existing approaches struggle when accuracy depends on identifying specific interface elements. These MLLMs also remain large and cannot adapt their reasoning depth to the task at hand. In this work, we introduce iSHIFT: Implicit Slow-fast Hybrid Inference with Flexible Tokens, a lightweight agent that integrates latent thinking (implicit chain-of-thought) with a perception control module. iSHIFT enables an MLLM to switch between a slow mode, which leverages detailed visual grounding for high precision and a fast mode that uses global cues for efficiency. Special perception tokens guide attention to relevant screen regions, allowing the model to decide both how to reason and where to focus. Despite its compact 2.5B size, iSHIFT matches state-of-the-art performance on multiple benchmark datasets.

Related papers

• AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent [21.148033135113927]
  We introduce an instruct-BLIP-based multimodal architecture that achieves superior performance in GUI automation.<n>We propose an adaptive feature renormalization-based (a token-level affine transformation) technique that effectively enriches low-resolution image embeddings.<n>We evaluate AFRAgent on Meta-GUI and AITW benchmarks, establishing a new state-of-the-art baseline for smartphone automation.
  arXiv  Detail & Related papers  (2025-11-30T11:32:54Z)
• Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMs [61.64185573373394]
  We propose a training-free framework that uses an MLLM's intrinsic uncertainty as a proactive guidance signal.<n>We introduce a unified mechanism that scores candidate visual inputs by response uncertainty, enabling the model to autonomously focus on the most salient data.<n>Our work validates that harnessing intrinsic uncertainty is a powerful, general strategy for enhancing fine-grained multimodal performance.
  arXiv  Detail & Related papers  (2025-10-01T09:20:51Z)
• Structuring GUI Elements through Vision Language Models: Towards Action Space Generation [43.932266242034025]
  Multimodal large language models (MLLMs) have emerged as pivotal tools in enhancing human-computer interaction.<n>This paper focuses on the application of MLLMs in the field of graphical user interface (GUI) elements structuring.<n>We introduce an IoU-Augmented Maximum Likelihood (IAML) training paradigm to bolster visual module capabilities.
  arXiv  Detail & Related papers  (2025-08-22T10:14:15Z)
• Less is More: Empowering GUI Agent with Context-Aware Simplification [62.02157661751793]
  We propose a context-aware framework for building an efficient and effective GUI Agent, termed SimpAgent.<n>With the above components, SimpAgent reduces 27% FLOPs and achieves superior GUI navigation performances.
  arXiv  Detail & Related papers  (2025-07-04T17:37:15Z)
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
  AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T07:30:29Z)
• ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction [15.220300812671494]
  We introduce ScreenLLM, a set of multimodal large language models (MLLMs) tailored for advanced UI understanding and action prediction.<n>Our work lays the foundation for scalable, robust, and intelligent GUI agents that enhance user interaction in diverse software environments.
  arXiv  Detail & Related papers  (2025-03-26T20:41:24Z)
• Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems [57.30711059396246]
  Current Graphical User Interface (GUI) grounding systems locate interface elements based on natural language instructions.<n>Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis.
  arXiv  Detail & Related papers  (2025-03-09T06:14:17Z)
• Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining [67.87810796668981]
  Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL)<n>Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations.<n>These improvements translate to significant gains in both web and OS agent downstream tasks.
  arXiv  Detail & Related papers  (2024-12-13T18:40:10Z)
• AppAgent v2: Advanced Agent for Flexible Mobile Interactions [57.98933460388985]
  This work introduces a novel LLM-based multimodal agent framework for mobile devices.<n>Our agent constructs a flexible action space that enhances adaptability across various applications.<n>Our results demonstrate the framework's superior performance, confirming its effectiveness in real-world scenarios.
  arXiv  Detail & Related papers  (2024-08-05T06:31:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.