CaptchaAgent: Verifiable Visual Reasoning for CAPTCHA Solving
Abstract Overview
CaptchaAgent is a structured visual reasoning framework for interactive CAPTCHA solving that inserts an explicit visual intermediate representation between page observation and action prediction. Instead of mapping screenshots directly to clicks or drags, it builds an ASCII-based representation that records entities, attributes, and spatial or topological relations, then decodes this representation into executable browser actions. The system further revisits ambiguous local regions at higher effective resolution and verifies candidate actions through task-preserving counterfactual transformations before execution. The paper evaluates this controller on online Open CaptchaWorld tasks and the offline Oedipus-data benchmark to examine both performance and failure modes.
Novelty
The paper’s main distinction is the use of a count-committed ASCII visual intermediate representation for CAPTCHA solving, combined with localized evidence refinement and counterfactual consistency verification at the action level. This design emphasizes an inspectable intermediate scene state rather than direct end-to-end action prediction or unconstrained chain-of-thought reasoning.
Results
On Open CaptchaWorld, CaptchaAgent improves Pass@1 by 21.3–25.8 percentage points over matched BU-Agent configurations across GPT-4.1, Gemini-2.5 Pro, and Claude-3.7-Sonnet, reaching 46.8% with Gemini-2.5 Pro. On Oedipus-data, it improves average Pass@1 by 9.2–12.3 points over the reported Oedipus solver across three matched backbones. Ablation results show that removing the visual intermediate representation causes the largest drop, while local refinement and consistency checking provide additional complementary gains.
Key Points
- CaptchaAgent converts CAPTCHA scenes into an explicit ASCII visual intermediate representation that commits to object counts, attributes, and relations before action generation.
- The method combines global structured parsing with localized evidence refinement and task-preserving counterfactual verification to improve action reliability.
- Experiments show consistent gains over matched baselines, while traces indicate that exact counting, continuous geometry, and ordered execution remain major failure sources.