CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training

• URL: http://arxiv.org/abs/2603.23559v1
• Date: Mon, 23 Mar 2026 23:35:59 GMT
• Title: CAPTCHA Solving for Native GUI Agents: Automated Reasoning-Action Data Generation and Self-Corrective Training
• Abstract summary: ReCAP is a CAPTCHA-capable native GUI agent that can robustly solve modern, interactive CAPTCHA challenges.<n>We develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories.<n>As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data.<n>Across held-out test sets, ReCAP improves CAPTCHA-solving success from roughly 30% to 80%, while maintaining strong performance on general GUI-agent benchmarks.
• Score: 18.35670595652216
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GUI agents are rapidly shifting from multi-module pipelines to end-to-end, native vision-language models (VLMs) that perceive raw screenshots and directly interact with digital devices. Despite rapid progress on general GUI tasks, CAPTCHA solving remains a major challenge. On the other hand, although specialized CAPTCHA solving pipelines exist, they cannot handle general GUI tasks. To address this gap, we introduce ReCAP: a CAPTCHA-capable native GUI agent that can robustly solve modern, interactive CAPTCHA challenges, while preserving their performance as a general GUI agent. We first develop a dynamic CAPTCHA system spanning seven representative CAPTCHA types, designed to stress primitive and complementary capabilities for CAPTCHA solving (e.g., robust OCR under heavy noise and text stylization, fine-grained visual understanding, and precise control). Then, we develop an automated data collection and curation pipeline that generates large-scale CAPTCHA interaction trajectories paired with reasoning traces. As CAPTCHA solving often requires multi-step interaction and recovery from intermediate mistakes, we further leverage failed trajectories to construct self-correction data, training agents to reflect on errors and correct their actions online. Across held-out test sets, ReCAP improves CAPTCHA-solving success from roughly 30\% to 80\%, while maintaining strong performance on general GUI-agent benchmarks.

Related papers

• CaptchaMind: Training CAPTCHA Solvers via Reinforcement Learning with Explicit Reasoning Supervision [50.35579781554922]
  We introduce Captcha-based Captcha-based RLMind, achieving 82.9% average success rate across eight tasks and 72.9% on real-world instances.<n>We present RLMind, an RLMind solver trained explicit reasoning process supervision, achieving 82.9% average success rate across eight tasks and 72.9% on real-world instances.
  arXiv  Detail & Related papers  (2026-05-19T08:38:44Z)
• Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defense [39.68941971572086]
  We introduce Next-Gen CAPTCHAs, a scalable defense framework to secure the next-generation web against advanced agents.<n>Unlike static datasets, our benchmark is built upon a robust data generation pipeline.<n>We exploit the persistent human-agent "Cognitive Gap" in interactive perception, memory, decision-making, and action.
  arXiv  Detail & Related papers  (2026-02-09T18:55:33Z)
• NGCaptcha: A CAPTCHA Bridging the Past and the Future [12.897322340526769]
  We introduce NGCAPTCHA, a Next Generation CAPTCHA framework that integrates a lightweight client side proof of work (PoW) mechanism with an AI resistant visual recognition challenge.<n>In NGCAPTCHA, a browser must first complete a small hash based PoW before any challenge is displayed.<n>Once the PoW is solved, the user is presented with a human friendly yet model resistant image selection task.
  arXiv  Detail & Related papers  (2025-12-18T06:14:54Z)
• Aura-CAPTCHA: A Reinforcement Learning and GAN-Enhanced Multi-Modal CAPTCHA System [1.4305544869388402]
  Aura-CAPTCHA was developed as a multi-modal CAPTCHA system to address vulnerabilities in traditional methods.<n>The design integrated Generative Adrial Networks (GANs) for generating dynamic image challenges, Reinforcement Learning (RL) for adaptive difficulty tuning, and Large Language Models (LLMs) for creating text and audio prompts.
  arXiv  Detail & Related papers  (2025-08-20T18:00:08Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning [82.42421823672954]
  AgentCPM-GUI is built for robust and efficient on-device GUI interaction.<n>Our training pipeline includes grounding-aware pre-training to enhance perception.<n>AgentCPM-GUI achieves state-of-the-art performance on five public benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T07:30:29Z)
• Oedipus: LLM-enchanced Reasoning CAPTCHA Solver [17.074422329618212]
  Oedipus is an innovative end-to-end framework for automated reasoning CAPTCHA solving. Central to this framework is a novel strategy that dissects the complex and human-easy-AI-hard tasks into a sequence of simpler and AI-easy steps. Our evaluation shows that Oedipus effectively resolves the studied CAPTCHAs, achieving an average success rate of 63.5%.
  arXiv  Detail & Related papers  (2024-05-13T06:32:57Z)
• CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automation [61.68049335444254]
  Multimodal large language models (MLLMs) have shown remarkable potential as human-like autonomous language agents to interact with real-world environments. We propose a Comprehensive Cognitive LLM Agent, CoCo-Agent, with two novel approaches, comprehensive environment perception (CEP) and conditional action prediction (CAP) With our technical design, our agent achieves new state-of-the-art performance on AITW and META-GUI benchmarks, showing promising abilities in realistic scenarios.
  arXiv  Detail & Related papers  (2024-02-19T08:29:03Z)
• A Survey of Adversarial CAPTCHAs on its History, Classification and Generation [69.36242543069123]
  We extend the definition of adversarial CAPTCHAs and propose a classification method for adversarial CAPTCHAs. Also, we analyze some defense methods that can be used to defend adversarial CAPTCHAs, indicating potential threats to adversarial CAPTCHAs.
  arXiv  Detail & Related papers  (2023-11-22T08:44:58Z)
• Robust Text CAPTCHAs Using Adversarial Examples [129.29523847765952]
  We propose a user-friendly text-based CAPTCHA generation method named Robust Text CAPTCHA (RTC) At the first stage, the foregrounds and backgrounds are constructed with randomly sampled font and background images. At the second stage, we apply a highly transferable adversarial attack for text CAPTCHAs to better obstruct CAPTCHA solvers.
  arXiv  Detail & Related papers  (2021-01-07T11:03:07Z)
• Deep-CAPTCHA: a deep learning based CAPTCHA solver for vulnerability assessment [1.027974860479791]
  This research investigates the weaknesses and vulnerabilities of the CAPTCHA generator systems. We develop a Convolutional Neural Network called Deep-CAPTCHA to achieve this goal. Our network's cracking accuracy leads to a high rate of 98.94% and 98.31% for the numerical and the alpha-numerical test datasets.
  arXiv  Detail & Related papers  (2020-06-15T11:44:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.