Beyond Black-Box Labels: Interpretable Criteria for Diagnosing SubjectiveNLP Tasks

• URL: http://arxiv.org/abs/2604.17022v1
• Date: Sat, 18 Apr 2026 15:09:20 GMT
• Title: Beyond Black-Box Labels: Interpretable Criteria for Diagnosing SubjectiveNLP Tasks
• Abstract summary: We propose a emphschema-level diagnostic for auditing expert-designed annotation schemas.<n>The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories.<n>These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a \emph{schema-level diagnostic} for auditing expert-designed annotation schemas \emph{prior to} gold-label commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hard-to-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidence-based audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm.

Related papers

• The Latent Diagnostic Taxonomy: A Framework for Constructing Classifiers and Diagnosing Their Decisions, Applied to Prompt Injection Detection [0.0]
  This paper proposes a framework for constructing a classifier as a safeguard layer, and for developing a complementary diagnostic.<n>Applying the framework to a public prompt injection dataset, we find that a substantial fraction of its confident decisions are not robust to removing a single token.<n>For each zone of the taxonomy, we also recommend strategies for remediating diagnosed prompts.
  arXiv  Detail & Related papers  (2026-08-26T21:55:15Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement [66.67463557274358]
  Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes.<n>We introduce Confidence-guided Concept Mining (CBCM) to identify reliable spurious attributes without attribute annotations.<n>We propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches.
  arXiv  Detail & Related papers  (2026-06-23T05:28:47Z)
• DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark [48.84943754804533]
  Multimodal documents contain diverse elements, such as tables, figures, and layouts.<n>Current approaches typically combine dense visual embedding models with supervised rerankers to achieve high-precision retrieval.<n>We propose DocRetriever, a plug-and-play framework that enhances visual retrieval via a layout-aware sparse embedding technique.
  arXiv  Detail & Related papers  (2026-05-28T14:50:53Z)
• Domain-Aware Hierarchical Contrastive Learning for Semi-Supervised Generalization Fault Diagnosis [49.89578655034085]
  We propose a unified framework termed domain-aware hierarchical contrastive learning (DAHCL) for fault diagnosis.<n>DAHCL captures source-domain geometric characteristics and calibrates pseudo-label predictions across heterogeneous source domains.<n>We also develop a hierarchical contrastive learning module that combines dynamic confidence stratification with fuzzy contrastive supervision.
  arXiv  Detail & Related papers  (2026-04-22T08:41:27Z)
• xList-Hate: A Checklist-Based Framework for Interpretable and Generalizable Hate Speech Detection [2.647843453311735]
  We introduce xList-Hate, a diagnostic framework that decomposes hate speech detection into a checklist of explicit, concept-level questions.<n>The diagnostic signals are aggregated by a lightweight, fully interpretable decision tree, yielding transparent and auditable predictions.<n>Our results suggest that reframing hate speech detection as a diagnostic reasoning task, rather than a monolithic classification problem.
  arXiv  Detail & Related papers  (2026-02-05T16:51:56Z)
• Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following [99.20581206115979]
  Multi-Crit is a benchmark for evaluating multimodal judges on their capacity to follow pluralistic criteria and produce reliable criterion-level judgments.<n> Comprehensive analysis of 25 LMMs reveals that 1) proprietary models still struggle to maintain consistent adherence to pluralistic criteria--especially in open-ended evaluation; 2) open-source models lag further behind in flexibly following diverse criteria; and 3) critic fine-tuning with holistic judgment signals enhances visual grounding but fails to generalize to pluralistic criterion-level judgment.
  arXiv  Detail & Related papers  (2025-11-26T18:35:17Z)
• CLASH: A Benchmark for Cross-Modal Contradiction Detection [15.134491772506196]
  CLASH is a novel benchmark for multimodal contradiction detection.<n>It features COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions.
  arXiv  Detail & Related papers  (2025-11-24T15:09:07Z)
• Rethinking Consistent Multi-Label Classification under Inexact Supervision [60.79309683889278]
  In partial multi-label learning, each instance is annotated with a candidate label set, among which only some labels are relevant.<n>In complementary multi-label learning, each instance is annotated with complementary labels indicating the classes to which the instance does not belong.
  arXiv  Detail & Related papers  (2025-10-05T08:30:32Z)
• When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search [0.0]
  Large language models (LLMs) are increasingly used to assign document relevance labels in information retrieval pipelines.<n>LLMs often disagree on borderline cases, raising concerns about how such disagreement affects downstream retrieval.<n>We show that model disagreement is systematic, not random.<n>We propose using classification disagreement as an object of analysis in retrieval evaluation, particularly in policy-relevant or thematic search tasks.
  arXiv  Detail & Related papers  (2025-07-02T20:53:51Z)
• Subjective Logic Encodings [20.458601113219697]
  Data perspectivism seeks to leverage inter-annotator disagreement to learn models.<n>Subjective Logic SLEs is a framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators.
  arXiv  Detail & Related papers  (2025-02-17T15:14:10Z)
• Robust Representation Learning for Unreliable Partial Label Learning [86.909511808373]
  Partial Label Learning (PLL) is a type of weakly supervised learning where each training instance is assigned a set of candidate labels, but only one label is the ground-truth. This is known as Unreliable Partial Label Learning (UPLL) that introduces an additional complexity due to the inherent unreliability and ambiguity of partial labels. We propose the Unreliability-Robust Representation Learning framework (URRL) that leverages unreliability-robust contrastive learning to help the model fortify against unreliable partial labels effectively.
  arXiv  Detail & Related papers  (2023-08-31T13:37:28Z)
• Ambiguity-Resistant Semi-Supervised Learning for Dense Object Detection [98.66771688028426]
  We propose a Ambiguity-Resistant Semi-supervised Learning (ARSL) for one-stage detectors. Joint-Confidence Estimation (JCE) is proposed to quantifies the classification and localization quality of pseudo labels. ARSL effectively mitigates the ambiguities and achieves state-of-the-art SSOD performance on MS COCO and PASCAL VOC.
  arXiv  Detail & Related papers  (2023-03-27T07:46:58Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• Classification with Rejection Based on Cost-sensitive Classification [83.50402803131412]
  We propose a novel method of classification with rejection by ensemble of learning. Experimental results demonstrate the usefulness of our proposed approach in clean, noisy, and positive-unlabeled classification.
  arXiv  Detail & Related papers  (2020-10-22T14:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.