SpiderSapien: Client-Centric Web Crawler and Security Scanner

• URL: http://arxiv.org/abs/2609.02532v1
• Date: Wed, 02 Sep 2026 12:42:39 GMT
• Title: SpiderSapien: Client-Centric Web Crawler and Security Scanner
• Abstract summary: We propose SpiderSapien, a client-centric crawler and security scanner.<n>SpiderSapien incorporates a unique combination of high-level, user-facing feedback channels from the web application.<n>In doing so, we demonstrate how to reliably discover and test deep states of modern web applications.
• Score: 11.836785827048379
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Black-box web application crawling and scanning play an important role for security testing of web applications. Yet state-of-the-art scanners fall short of addressing key characteristics of a modern web application: its extreme dynamism and interactivity on the client side. This paper identifies immersive interaction as a key ingredient for scanners to deeply explore modern web applications. We propose SpiderSapien, a client-centric crawler and security scanner. SpiderSapien incorporates a unique combination of high-level, user-facing feedback channels from the web application to achieve immersive interaction in a black-box crawling loop. These feedback channels include both novel methods to detect interactable elements and sensibly order UI interactions, and orthogonally using an LLM to solve forms. In doing so, we demonstrate how to reliably discover and test deep states of modern web applications. Furthermore, our modular approach and useful abstraction layer can serve as a building block for future scanners. The evaluation of our approach shows substantial improvements in both code coverage and vulnerability detection over previous work. Our approach increased average code coverage across applications by at least 46% over any other scanner, or 16% when compared to the union of all other scanners. We find XSS vulnerabilities in 7 web applications, while any other scanner finds XSS in up to 2 applications.

Related papers

• BrowserForge: Scaling Web Episode via Parallel Browser Sandboxes [54.96895469970198]
  BrowserForge generates web interaction data at scale by driving many browser sandboxes in parallel over the open web.<n> BrowserForge couples three components: an open-web sourcing stage that exposes the agent to hundreds of thousands of real, openly reachable websites.<n>A rule-plus-model cleaning pipeline removes failed runs and rewrites the surviving reasoning into a single unified chain-of-thought style.
  arXiv  Detail & Related papers  (2026-08-25T17:35:42Z)
• Whose Agent Are You? Multi-Layer Fingerprinting and Attribution of Autonomous Web Agents [14.225575847186592]
  We show that AI web agents can be effectively distinguished from humans and traditional crawlers using a multi-layer fingerprint.<n>We implement this mechanism as a programmatic logging framework that can be deployed on a live, instrumented domain.<n>Our findings demonstrate that cross-layer agent tracking provides a robust, evasion-resistant strategy for content protection and web security policy enforcement.
  arXiv  Detail & Related papers  (2026-06-18T20:07:39Z)
• Nested Browser-Use Learning for Agentic Information Seeking [60.775556172513014]
  Information-seeking (IS) agents have achieved strong performance across a range of wide and deep search tasks, yet their tool use remains largely restricted to API-level snippet retrieval and URL-based page fetching.<n>We propose Nested Browser-Use Learning (NestBrowse), which introduces a minimal and complete browser-action framework that decouples interaction control from page exploration through a nested structure.
  arXiv  Detail & Related papers  (2025-12-29T17:59:14Z)
• BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions [48.194688161526756]
  BrowserAgent operates directly on raw web pages via Playwright through a set of predefined browser actions.<n>We introduce an explicit memory mechanism to store key conclusions across steps, further enhancing the model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-12T15:43:37Z)
• BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Tasks [51.803138848305814]
  We introduce BrowserArena, a live open-web agent evaluation platform that collects user-submitted tasks.<n>We identify three consistent failure modes: captcha resolution, pop-up banner removal, and direct navigation to URLs.<n>Our findings surface both the diversity and brittleness of current web agents.
  arXiv  Detail & Related papers  (2025-10-02T15:22:21Z)
• Unlocking User-oriented Pages: Intention-driven Black-box Scanner for Real-world Web Applications [16.223807733708767]
  Hoyen is a black-box scanner that uses the Large Language Model to predict user intention.<n> Hoyen has been rigorously evaluated on 12 popular open-source web applications and compared with 6 representative tools.
  arXiv  Detail & Related papers  (2025-04-29T14:14:30Z)
• Enhancing Webshell Detection With Deep Learning-Powered Methods [0.6390468088226495]
  Webshell attacks are becoming more common, requiring robust detection mechanisms to protect web applications.<n>The dissertation proposes ASAF, an advanced DL-Powered Source-Code Scanning Framework that uses signature-based methods and deep learning algorithms to detect known and unknown webshells.<n>Second, the dissertation introduces a deep neural network that detects webshells using real-time HTTP traffic analysis of web applications.
  arXiv  Detail & Related papers  (2024-12-07T04:26:36Z)
• Beyond Browsing: API-Based Web Agents [58.39129004543844]
  API-Based Agents outperform web Browsing Agents in experiments on WebArena.<n>Hybrid Agents out-perform both others nearly uniformly across tasks.<n>Results strongly suggest that when APIs are available, they present an attractive alternative to relying on web browsing alone.
  arXiv  Detail & Related papers  (2024-10-21T19:46:06Z)
• Dissecting Adversarial Robustness of Multimodal LM Agents [70.2077308846307]
  We manually create 200 targeted adversarial tasks and evaluation scripts in a realistic threat model on top of VisualWebArena.<n>We find that we can successfully break latest agents that use black-box frontier LMs, including those that perform reflection and tree search.<n>We also use ARE to rigorously evaluate how the robustness changes as new components are added.
  arXiv  Detail & Related papers  (2024-06-18T17:32:48Z)
• CRATOR: a Dark Web Crawler [1.7224362150588657]
  This study proposes a general dark web crawler designed to extract pages handling security protocols, such as captchas. Our approach uses a combination of seed URL lists, link analysis, and scanning to discover new content.
  arXiv  Detail & Related papers  (2024-05-10T09:39:12Z)
• AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation [54.17246674188208]
  Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website. We introduce the paradigm of generating web scrapers with large language models (LLMs) and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently.
  arXiv  Detail & Related papers  (2024-04-19T09:59:44Z)
• Locality-Sensitive Hashing for Efficient Web Application Security Testing [0.0]
  We present a novel approach to detect redundant content for security testing purposes. The algorithm applies locality-sensitive hashing using MinHash sketches in order to analyze the Document Object Model (DOM) structure of web pages. Our experimental results show that this approach allows a successful scan of RIAs that cannot be crawled otherwise.
  arXiv  Detail & Related papers  (2020-01-04T21:05:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.