PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection

• URL: http://arxiv.org/abs/2604.15776v1
• Date: Fri, 17 Apr 2026 07:32:46 GMT
• Title: PIIBench: A Unified Multi-Source Benchmark Corpus for Personally Identifiable Information Detection
• Abstract summary: PIIBench is a unified benchmark corpus for Personally Identifiable Information (PII) detection in natural language text.<n>We consolidate ten publicly available datasets spanning synthetic PII corpora, multilingual Named Entity Recognition (NER) benchmarks, and financial domain annotated text.<n>We produce a corpus of 2,369,883 annotated sequences and 3.35 million entity mentions across 48 canonical PII entity types.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present PIIBench, a unified benchmark corpus for Personally Identifiable Information (PII) detection in natural language text. Existing resources for PII detection are fragmented across domain-specific corpora with mutually incompatible annotation schemes, preventing systematic comparison of detection systems. We consolidate ten publicly available datasets spanning synthetic PII corpora, multilingual Named Entity Recognition (NER) benchmarks, and financial domain annotated text, yielding a corpus of 2,369,883 annotated sequences and 3.35 million entity mentions across 48 canonical PII entity types. We develop a principled normalization pipeline that maps 80+ source-specific label variants to a standardized BIO tagging scheme, applies frequency-based suppression of near absent entity types, and produces stratified 80/10/10 train/validation/test splits preserving source distribution. To establish baseline difficulty, we evaluate eight published systems spanning rule-based engines (Microsoft Presidio), general purpose NER models (spaCy, BERT-base NER, XLM-RoBERTa NER, SpanMarker mBERT, SpanMarker BERT), a PII-specific model (Piiranha DeBERTa), and a financial NER specialist (XtremeDistil FiNER). All systems achieve span-level F1 below 0.14, with the best system (Presidio, F1=0.1385) still producing zero recall on most entity types. These results directly quantify the domain-silo problem and demonstrate that PIIBench presents a substantially harder and more comprehensive evaluation challenge than any existing single source PII dataset. The dataset construction pipeline and benchmark evaluation code are publicly available at https://github.com/pritesh-2711/pii-bench.

Related papers

• Mind the Gap: Robustness Risks in PII Detection Systems [6.167318681955322]
  We study how different PII detection paradigms fail under noisy, unstructured, and informal inputs.<n>No single architecture is uniformly reliable across PII categories.<n>Motivated by these findings, we propose a hybrid detection pipeline with a QA-driven feedback loop for iterative risk mitigation.
  arXiv  Detail & Related papers  (2026-09-03T07:23:28Z)
• Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa [0.0]
  PII detection systems are frequently trained within narrow source or domain boundaries, limiting coverage when deployed on heterogeneous text.<n>We study model fine-tuning on a corrected multi-source PIIBench preparation spanning 82 retained entity types across ten source datasets.<n>We evaluate three DeBERTa-based approaches: direct token classification fine-tuning, a source-conditioned hierarchical model (SC+H), and a three-phase curriculum extension (SC+H+Curr)
  arXiv  Detail & Related papers  (2026-05-25T13:12:42Z)
• Multi-Level Contextual Token Relation Modeling for Machine-Generated Text Detection [105.1403233464793]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, underscoring the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods.<n>We propose a multi-level contextual token relation modeling framework for MGT detection.
  arXiv  Detail & Related papers  (2026-05-15T15:55:58Z)
• GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction [6.46529664618265]
  We present GLiNER2-PII, a small 0.3B- parameter model adapted from GLiNER2 to recognize a broad taxonomy of 42 PII entity types at character-span resolution.<n>Training such systems is constrained by the scarcity of shareable annotated data and the privacy risks associated with collecting real PII at scale.<n>We release the model on Hugging Face to support further research and practical deployment of open PII detection systems.
  arXiv  Detail & Related papers  (2026-05-11T04:29:30Z)
• BRIDGE and TCH-Net: Heterogeneous Benchmark and Multi-Branch Baseline for Cross-Domain IoT Botnet Detection [1.2019888796331233]
  IoT botnet detection has advanced, yet most published systems are validated on a single dataset and rarely generalise across environments.<n>We introduce BRIDGE, the first formally specified heterogeneous multi-dataset benchmark for IoT intrusion detection.<n>We establish the first community generalisation baseline at mean LODO F1 = 0.5577, a result that shifts the agenda from single-benchmark optimisation to cross-environment generalisation.
  arXiv  Detail & Related papers  (2026-04-13T11:25:39Z)
• Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement [1.8429656136522097]
  We present a comprehensive approach for multiword expression (MWE) identification that combines binary token-level classification, linguistic feature integration, and data augmentation.<n>Our DeBERTa-v3-large model achieves 69.8% F1 on the CoAM dataset, surpassing the best results (Qwen-72B, 57.8% F1) on this dataset by 12 points while using 165x fewer parameters.
  arXiv  Detail & Related papers  (2026-01-27T08:42:54Z)
• An Evaluation Study of Hybrid Methods for Multilingual PII Detection [0.026059379504241156]
  We present RECAP, a framework that combines deterministic regular expressions with context-aware large language models (LLMs) for scalable PII detection.<n>Our system outperforms fine-tuned NER models by 82% and zero-shot LLMs by 17% in weighted F1-score.<n>This work offers a scalable and adaptable solution for efficient PII detection in compliance-focused applications.
  arXiv  Detail & Related papers  (2025-10-08T21:03:59Z)
• MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection [84.75972919995398]
  This paper presents a multi-agent system that uses relation extraction to detect disinformation in news articles.<n>The proposed Agentic AI system combines four agents: (i) a machine learning agent (logistic regression), (ii) a Wikipedia knowledge check agent, and (iv) a web-scraped data analyzer.<n>Results demonstrate that the multi-agent ensemble achieves 95.3% accuracy with an F1 score of 0.964, significantly outperforming individual agents and traditional approaches.
  arXiv  Detail & Related papers  (2025-08-13T19:14:48Z)
• ForensicHub: A Unified Benchmark & Codebase for All-Domain Fake Image Detection and Localization [54.64604722756443]
  ForensicHub is the first unified benchmark for all-domain fake image detection and localization.<n>It decomposes forensic pipelines into interchangeable components across datasets, transforms, models, and evaluators.<n>It offers 8 key actionable insights into FIDL model architecture, dataset characteristics, and evaluation standards.
  arXiv  Detail & Related papers  (2025-05-16T08:49:59Z)
• SNaRe: Domain-aware Data Generation for Low-Resource Event Detection [77.32937742071475]
  Event Detection is critical for enabling reasoning in highly specialized domains such as biomedicine, law, and epidemiology.<n>We introduce SNaRe, a domain-aware synthetic data generation framework composed of three components: Scout, Narrator, and Refiner.<n>Scout extracts triggers from unlabeled target domain data and curates a high-quality domain-specific trigger list.<n>Narrator, conditioned on these triggers, generates high-quality domain-aligned sentences, and Refiner identifies additional event mentions.
  arXiv  Detail & Related papers  (2025-02-24T18:20:42Z)
• Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypes [6.30817290125825]
  We introduce the GUS-Net Framework, comprising the GUS dataset and a multi-label token-level detector for span-level analysis of social bias.<n>The GUS dataset contains 3,739 unique snippets across multiple domains, with over 69,000 token-level annotations.<n>We formulate bias detection as multi-label token-level classification and benchmark both encoder-based models and decoder-based large language models.
  arXiv  Detail & Related papers  (2024-10-10T21:51:22Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Dual Refinement Feature Pyramid Networks for Object Detection [2.88935873409577]
  FPN is a common component used in object detectors, it supplements multi-scale information by adjacent level features and summation. In this paper, we analyze the design defects from pixel level and feature map level. We design a novel parameter-free feature pyramid networks named Dual Refinement Feature Pyramid Networks for the problems.
  arXiv  Detail & Related papers  (2020-12-03T07:17:03Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.