WebPII: Benchmarking Visual PII Detection for Computer-Use Agents

• URL: http://arxiv.org/abs/2603.17357v1
• Date: Wed, 18 Mar 2026 04:41:16 GMT
• Title: WebPII: Benchmarking Visual PII Detection for Computer-Use Agents
• Abstract summary: We introduce WebPII, a fine-grained synthetic benchmark of 44,865 annotated e-commerce UI images.<n>We train WebRedact to demonstrate practical utility, more than doubling text-extraction baseline accuracy.<n>We release the dataset and model to support privacy-preserving computer use research.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computer use agents create new privacy risks: training data collected from real websites inevitably contains sensitive information, and cloud-hosted inference exposes user screenshots. Detecting personally identifiable information in web screenshots is critical for privacy-preserving deployment, but no public benchmark exists for this task. We introduce WebPII, a fine-grained synthetic benchmark of 44,865 annotated e-commerce UI images designed with three key properties: extended PII taxonomy including transaction-level identifiers that enable reidentification, anticipatory detection for partially-filled forms where users are actively entering data, and scalable generation through VLM-based UI reproduction. Experiments validate that these design choices improve layout-invariant detection across diverse interfaces and generalization to held-out page types. We train WebRedact to demonstrate practical utility, more than doubling text-extraction baseline accuracy (0.753 vs 0.357 mAP@50) at real-time CPU latency (20ms). We release the dataset and model to support privacy-preserving computer use research.

Related papers

• PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails [42.825314356642956]
  PersonaTrail is a benchmark for personalized web agents operating in a managed open web environment.<n>PACMem is a framework that decomposes raw browsing histories into two types of structured memory.<n>Experiments show that PACMem consistently outperforms existing memory-based baselines on both tasks.
  arXiv  Detail & Related papers  (2026-05-30T13:27:55Z)
• LiveWeb-IE: A Benchmark For Online Web Information Extraction [48.82654261583883]
  Web information extraction (WIE) is the task of automatically extracting data from web pages, offering high utility for various applications.<n>We introduce dataset, a new benchmark designed for evaluating WIE systems directly against live websites.<n>We also propose Visual Grounding Scraper (VGS), a novel multi-stage agentic framework that mimics human cognitive processes by visually narrowing down web page content to extract desired information.
  arXiv  Detail & Related papers  (2026-03-14T05:55:11Z)
• Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible [12.742325129012576]
  Mobile Graphical User Interface (GUI) agents have demonstrated strong capabilities in automating complex smartphone tasks.<n>We propose anonymization-based privacy protection framework that enforces the principle of available-but-invisible access to sensitive data.<n>Our system detects sensitive UI content using a PII-aware recognition model and replaces it with deterministic, type-preserving placeholders.
  arXiv  Detail & Related papers  (2026-02-08T15:50:04Z)
• A Dual-stage Prompt-driven Privacy-preserving Paradigm for Person Re-Identification [42.56589115173974]
  We propose a Dual-stage Prompt-driven Privacy-preserving Paradigm (DPPP)<n>In the first stage, we generate rich prompts incorporating multi-dimensional attributes that drive the diffusion model to synthesize diverse data end-to-end.<n>In the second stage, we propose a Prompt-driven Disentanglement Mechanism (PDM) to learn domain-invariant generalization features.
  arXiv  Detail & Related papers  (2025-11-07T09:17:48Z)
• Leveraging Scene Context with Dual Networks for Sequential User Behavior Modeling [58.72480539725212]
  We propose a novel Dual Sequence Prediction networks (DSPnet) to capture the dynamic interests and interplay between scenes and items for future behavior prediction.<n>DSPnet consists of two parallel networks dedicated to learn users' dynamic interests over items and scenes, and a sequence feature enhancement module to capture the interplay for enhanced future behavior prediction.
  arXiv  Detail & Related papers  (2025-09-30T12:26:57Z)
• Following the Clues: Experiments on Person Re-ID using Cross-Modal Intelligence [0.0]
  CRID is a cross-modal framework combining Large Vision-Language Models, Graph Attention Networks, and representation learning.<n>Our approach focuses on identifying and leveraging interpretable features, enabling the detection of semantically meaningful PII beyond low-level appearance cues.<n>Our experiments show improved performance in practical cross-dataset Re-ID scenarios.
  arXiv  Detail & Related papers  (2025-07-02T09:10:33Z)
• Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis [57.371814877372515]
  Graphical user interface (GUI) grounding remains a critical bottleneck in computer use agent development.<n>We introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types.<n>We synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples.
  arXiv  Detail & Related papers  (2025-05-19T15:09:23Z)
• AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials [53.376263056033046]
  Existing approaches rely on expensive human annotation, making them unsustainable at scale.<n>We propose AgentTrek, a scalable data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials.<n>Our fully automated approach significantly reduces data collection costs, achieving a cost of just $0.55 per high-quality trajectory without human annotators.
  arXiv  Detail & Related papers  (2024-12-12T18:59:27Z)
• Federated Face Forgery Detection Learning with Personalized Representation [63.90408023506508]
  Deep generator technology can produce high-quality fake videos that are indistinguishable, posing a serious social threat. Traditional forgery detection methods directly centralized training on data. The paper proposes a novel federated face forgery detection learning with personalized representation.
  arXiv  Detail & Related papers  (2024-06-17T02:20:30Z)
• FP-Fed: Privacy-Preserving Federated Detection of Browser Fingerprinting [10.671588861099323]
  Browser fingerprinting provides an attractive alternative to third-party cookies for tracking users across the web. Previous work proposed several techniques to detect its prevalence and severity. We present FP-Fed, the first distributed system for browser fingerprinting detection.
  arXiv  Detail & Related papers  (2023-11-28T16:43:17Z)
• Federated Test-Time Adaptive Face Presentation Attack Detection with Dual-Phase Privacy Preservation [100.69458267888962]
  Face presentation attack detection (fPAD) plays a critical role in the modern face recognition pipeline. Due to legal and privacy issues, training data (real face images and spoof images) are not allowed to be directly shared between different data sources. We propose a Federated Test-Time Adaptive Face Presentation Attack Detection with Dual-Phase Privacy Preservation framework.
  arXiv  Detail & Related papers  (2021-10-25T02:51:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.