Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification

• URL: http://arxiv.org/abs/2608.16928v1
• Date: Wed, 05 Aug 2026 05:00:41 GMT
• Title: Benchmarking Classical and Transformer-Based Models for Document Sensitivity Classification
• Abstract summary: This paper introduces Strategic 16K, a leakage-controlled corpus of 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD)<n>We document an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies.<n>Among classical models, TF-IDF with Logistic Regression achieves the strongest performance at significantly lower computational cost.
• Score: 1.096626056612224
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic sensitivity classification of organizational documents is a critical yet underserved problem, where the consequences of misclassification range from regulatory violations to security breaches. While AI-based approaches offer a scalable alternative to manual review, their reliability depends fundamentally on the integrity of training data. A pervasive but underreported problem in this domain is label leakage: residual classification markers embedded within document bodies that allow models to exploit surface shortcuts rather than learning genuine content-based sensitivity signals, producing performance estimates that are inflated and unreliable. This paper addresses this problem by introducing Strategic 16K, a carefully constructed, leakage-controlled corpus of 16,000 diplomatic cables sourced from the WikiLeaks Public Library of US Diplomacy (PlusD), and presents a systematic benchmark evaluating six model architectures spanning classical machine learning and transformer-based approaches. We document an extended leakage removal protocol that identifies and eliminates three categories of residual classification markers embedded within document bodies. On the clean benchmark, BERT achieves the strongest performance (Accuracy = 89.14%, F1 = 89.33%), followed by ELECTRA (Accuracy = 88.57%, F1 = 88.90%). Among classical models, TF-IDF with Logistic Regression achieves the strongest performance at significantly lower computational cost. These results constitute the first fully reproducible sensitivity classification benchmark constructed under explicit leakage-controlled conditions from WikiLeaks PlusD.

Related papers

• Security Document Classification with a Fine-Tuned Local Large Language Model: Benchmark Data and an Open-Source System [0.0]
  This study presents TorchSight, an open-source local system for security document classification built around a fine-tuned Qwen 3.5 27B model.<n>The model was trained on 78,358 samples from 13 permissively licensed sources and GPT-4 synthetic data covering seven security categories and 51 subcategories.<n>Results show that a fine-tuned local model can support accurate security document classification while keeping document processing under local control.
  arXiv  Detail & Related papers  (2026-05-19T18:18:19Z)
• GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees [1.6058099298620423]
  We introduce the emphGF-Score (GREAT-Fairness Score), a framework that decomposes the certified GREAT Score into per-class robustness profiles.<n>We find that the decomposition is exact, that per-class scores reveal consistent vulnerability patterns, and that more robust models tend to exhibit greater class-level disparity.<n>These results establish a practical, attack-free auditing pipeline for diagnosing where certified robustness guarantees fail to protect all classes equally.
  arXiv  Detail & Related papers  (2026-04-14T14:03:22Z)
• Learning to Prioritize IT Tickets: A Comparative Evaluation of Embedding-based Approaches and Fine-Tuned Transformer Models [0.0]
  Prioritizing service tickets in IT Service Management (ITSM) is critical for operational efficiency but remains challenging.<n>We evaluate two families of approaches for ticket prioritization: embedding-based pipelines that combine dimensionality reduction, clustering, and classical generalizations, and a fine-tuned multilingual transformer that processes both textual and numerical features.
  arXiv  Detail & Related papers  (2025-11-28T16:02:06Z)
• EReLiFM: Evidential Reliability-Aware Residual Flow Meta-Learning for Open-Set Domain Generalization under Noisy Labels [85.78886153628663]
  Open-Set Domain Generalization aims to enable deep learning models to recognize unseen categories in new domains.<n>Label noise hinders open-set domain generalization by corrupting source-domain knowledge.<n>We propose Evidential Reliability-Aware Residual Flow Meta-Learning (EReLiFM) to bridge domain gaps.
  arXiv  Detail & Related papers  (2025-10-14T16:23:11Z)
• LVLMs as inspectors: an agentic framework for category-level structural defect annotation [3.2445985501669434]
  A novel agentic annotation framework, Agent-based Defect Pattern Tagger, is introduced.<n>It integrates Large Vision-Language Models (LVLMs) with a semantic pattern matching module and an iterative self-questioning refinement mechanism.<n>It transforms raw visual data into high-quality, semantically labeled defect datasets without any manual supervision.
  arXiv  Detail & Related papers  (2025-10-01T07:31:42Z)
• Attribute Fusion-based Classifier on Framework of Belief Structure [46.24928730489845]
  Dempster-Shafer Theory (DST) provides a powerful framework for modeling uncertainty and has been widely applied to multi-attribute classification tasks.<n>Traditional DST-based attribute fusion-based classifiers suffer from oversimplified membership function modeling and limited exploitation of the belief structure brought by basic probability assignment (BPA)<n>This paper presents an enhanced attribute fusion-based classifier that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-08-31T09:05:15Z)
• GLiClass: Generalist Lightweight Model for Sequence Classification Tasks [49.2639069781367]
  We propose GLiClass, a novel method that adapts the GLiNER architecture for sequence classification tasks.<n>Our approach achieves strong accuracy and efficiency comparable to embedding-based methods, while maintaining the flexibility needed for zero-shot and few-shot learning scenarios.
  arXiv  Detail & Related papers  (2025-08-11T06:22:25Z)
• Graph-Structured Data Analysis of Component Failure in Autonomous Cargo Ships Based on Feature Fusion [20.287188044863925]
  This paper proposes a novel hybrid feature fusion framework for constructing a graph-structured dataset of failure modes.<n>A hierarchical feature fusion framework is constructed, using Word2Vec encoding to encode subsystem/component features, BERT-KPCA to process failure modes/reasons, and Sentence-BERT to quantify the semantic association between failure impact and emergency decision-making.
  arXiv  Detail & Related papers  (2025-07-18T08:02:49Z)
• Hybrid-Segmentor: A Hybrid Approach to Automated Fine-Grained Crack Segmentation in Civil Infrastructure [52.2025114590481]
  We introduce Hybrid-Segmentor, an encoder-decoder based approach that is capable of extracting both fine-grained local and global crack features. This allows the model to improve its generalization capabilities in distinguish various type of shapes, surfaces and sizes of cracks. The proposed model outperforms existing benchmark models across 5 quantitative metrics (accuracy 0.971, precision 0.804, recall 0.744, F1-score 0.770, and IoU score 0.630), achieving state-of-the-art status.
  arXiv  Detail & Related papers  (2024-09-04T16:47:16Z)
• Adaptive Hierarchical Certification for Segmentation using Randomized Smoothing [87.48628403354351]
  certification for machine learning is proving that no adversarial sample can evade a model within a range under certain conditions. Common certification methods for segmentation use a flat set of fine-grained classes, leading to high abstain rates due to model uncertainty. We propose a novel, more practical setting, which certifies pixels within a multi-level hierarchy, and adaptively relaxes the certification to a coarser level for unstable components.
  arXiv  Detail & Related papers  (2024-02-13T11:59:43Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection [85.53263670166304]
  One-stage detector basically formulates object detection as dense classification and localization. Recent trend for one-stage detectors is to introduce an individual prediction branch to estimate the quality of localization. This paper delves into the representations of the above three fundamental elements: quality estimation, classification and localization.
  arXiv  Detail & Related papers  (2020-06-08T07:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.