Label Forensics: Interpreting Hard Labels in Black-Box Text Classifier

• URL: http://arxiv.org/abs/2512.01514v1
• Date: Mon, 01 Dec 2025 10:39:51 GMT
• Title: Label Forensics: Interpreting Hard Labels in Black-Box Text Classifier
• Abstract summary: We introduce label forensics, a black-box framework that reconstructs a label's semantic meaning.<n> Experiments on multiple black-box classifiers achieve an average label consistency of around 92.24 percent.
• Score: 27.11428257480993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread adoption of natural language processing techniques has led to an unprecedented growth of text classifiers across the modern web. Yet many of these models circulate with their internal semantics undocumented or even intentionally withheld. Such opaque classifiers, which may expose only hard-label outputs, can operate in unregulated web environments or be repurposed for unknown intents, raising legitimate forensic and auditing concerns. In this paper, we position ourselves as investigators and work to infer the semantic concept each label encodes in an undocumented black-box classifier. Specifically, we introduce label forensics, a black-box framework that reconstructs a label's semantic meaning. Concretely, we represent a label by a sentence embedding distribution from which any sample reliably reflects the concept the classifier has implicitly learned for that label. We believe this distribution should maintain two key properties: precise, with samples consistently classified into the target label, and general, covering the label's broad semantic space. To realize this, we design a semantic neighborhood sampler and an iterative optimization procedure to select representative seed sentences that jointly maximize label consistency and distributional coverage. The final output, an optimized seed sentence set combined with the sampler, constitutes the empirical distribution representing the label's semantics. Experiments on multiple black-box classifiers achieve an average label consistency of around 92.24 percent, demonstrating that the embedding regions accurately capture each classifier's label semantics. We further validate our framework on an undocumented HuggingFace classifier, enabling fine-grained label interpretation and supporting responsible AI auditing.

Related papers

• One Size Does Not Fit All: Exploring Variable Thresholds for Distance-Based Multi-Label Text Classification [1.8165993946919816]
  Distance-based unsupervised text classification uses semantic similarity between a label and a text to determine label relevance.<n>Similarity distributions show statistically significant differences across models, datasets and even label sets.<n>We propose a novel method for optimizing label-specific thresholds using a validation set.
  arXiv  Detail & Related papers  (2025-10-13T08:52:14Z)
• Sanitizing Manufacturing Dataset Labels Using Vision-Language Models [1.0819408603463427]
  This paper introduces Vision-Language Sanitization and Refinement (VLSR), which is a vision-language-based framework for label sanitization and refinement.<n>The method embeds both images and their associated textual labels into a shared semantic space leveraging the CLIP vision-language model.<n> Experimental results demonstrate that the VLSR framework successfully identifies problematic labels and improves label consistency.
  arXiv  Detail & Related papers  (2025-06-30T02:13:09Z)
• Zero-Shot Pseudo Labels Generation Using SAM and CLIP for Semi-Supervised Semantic Segmentation [0.0]
  We propose a method to train a semantic segmentation model using images with annotated labels and pseudo labels.<n>The accuracy of the model depends on the quality of the pseudo labels and the amount of data with annotated labels.<n>The effectiveness of the proposed method is demonstrated through the experiments using the public datasets: PASCAL and MS COCO.
  arXiv  Detail & Related papers  (2025-05-26T11:31:13Z)
• Mixed Blessing: Class-Wise Embedding guided Instance-Dependent Partial Label Learning [53.64180787439527]
  In partial label learning (PLL), every sample is associated with a candidate label set comprising the ground-truth label and several noisy labels.<n>For the first time, we create class-wise embeddings for each sample, which allow us to explore the relationship of instance-dependent noisy labels.<n>To reduce the high label ambiguity, we introduce the concept of class prototypes containing global feature information.
  arXiv  Detail & Related papers  (2024-12-06T13:25:39Z)
• Scribble Hides Class: Promoting Scribble-Based Weakly-Supervised Semantic Segmentation with Its Class Label [16.745019028033518]
  We propose a class-driven scribble promotion network, which utilizes both scribble annotations and pseudo-labels informed by image-level classes and global semantics for supervision. Experiments on the ScribbleSup dataset with different qualities of scribble annotations outperform all the previous methods, demonstrating the superiority and robustness of our method.
  arXiv  Detail & Related papers  (2024-02-27T14:51:56Z)
• Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations [91.67511167969934]
  imprecise label learning (ILL) is a framework for the unification of learning with various imprecise label configurations. We demonstrate that ILL can seamlessly adapt to partial label learning, semi-supervised learning, noisy label learning, and, more importantly, a mixture of these settings.
  arXiv  Detail & Related papers  (2023-05-22T04:50:28Z)
• Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels [60.675714333081466]
  Multi-label recognition (MLR) with incomplete labels is very challenging. Recent works strive to explore the image-to-label correspondence in the vision-language model, ie, CLIP, to compensate for insufficient annotations. We advocate remedying the deficiency of label supervision for the MLR with incomplete labels by deriving a structured semantic prior.
  arXiv  Detail & Related papers  (2023-03-23T12:39:20Z)
• A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning [111.05365744744437]
  Unsupervised contrastive learning labels crops of the same image as positives, and other image crops as negatives. In this work, we first prove that for contrastive learning, inaccurate label assignment heavily impairs its generalization for semantic instance discrimination. Inspired by this theory, we propose a novel self-labeling refinement approach for contrastive learning.
  arXiv  Detail & Related papers  (2021-06-28T14:24:52Z)
• PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision [12.284208932393073]
  This paper presents a novel procedure for producing semantic segmentation from images given some point level annotations. We propose training a CNN that is normally fully supervised using our pseudo labels in place of ground truth labels. Our method achieves state of the art results for point supervised semantic segmentation on the PASCAL VOC 2012 dataset citeeveringham2010pascal, even outperforming state of the art methods for stronger bounding box and squiggle supervision.
  arXiv  Detail & Related papers  (2020-07-10T21:25:27Z)
• Debiased Contrastive Learning [64.98602526764599]
  We develop a debiased contrastive objective that corrects for the sampling of same-label datapoints. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2020-07-01T04:25:24Z)
• MatchGAN: A Self-Supervised Semi-Supervised Conditional Generative Adversarial Network [51.84251358009803]
  We present a novel self-supervised learning approach for conditional generative adversarial networks (GANs) under a semi-supervised setting. We perform augmentation by randomly sampling sensible labels from the label space of the few labelled examples available. Our method surpasses the baseline with only 20% of the labelled examples used to train the baseline.
  arXiv  Detail & Related papers  (2020-06-11T17:14:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.