Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference

• URL: http://arxiv.org/abs/2604.19943v1
• Date: Tue, 21 Apr 2026 19:45:16 GMT
• Title: Structured Disagreement in Health-Literacy Annotation: Epistemic Stability, Conceptual Difficulty, and Agreement-Stratified Inference
• Abstract summary: Perspectivist approaches treat disagreement as potentially informative rather than erroneous.<n>We present a large-scale analysis of graded health-literacy annotations from 6,323 responses collected in Ecuador and Peru.<n>Key social-scientific effects, including country, education, and urban-rural differences, vary in magnitude and in some cases reverse direction across levels of inter-annotator agreement.
• Score: 0.8252603363637361
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Annotation pipelines in Natural Language Processing (NLP) commonly assume a single latent ground truth per instance and resolve disagreement through label aggregation. Perspectivist approaches challenge this view by treating disagreement as potentially informative rather than erroneous. We present a large-scale analysis of graded health-literacy annotations from 6,323 open-ended COVID-19 responses collected in Ecuador and Peru. Each response was independently labeled by multiple annotators using proportional correctness scores, reflecting the degree to which responses align with normative public-health guidelines, allowing us to analyze the full distribution of judgments rather than aggregated labels. Variance decomposition shows that question-level conceptual difficulty accounts for substantially more variance than annotator identity, indicating that disagreement is structured by the task itself rather than driven by individual raters. Agreement-stratified analyses further reveal that key social-scientific effects, including country, education, and urban-rural differences, vary in magnitude and in some cases reverse direction across levels of inter-annotator agreement. These findings suggest that graded health-literacy evaluation contains both epistemically stable and unstable components, and that aggregating across them can obscure important inferential differences. We therefore argue that strong perspectivist modeling is not only conceptually justified but statistically necessary for valid inference in graded interpretive tasks.

Related papers

• Toward Automated Detection of Documentation Inconsistencies in Electronic Health Records [16.68744679025527]
  General-domain large language model (LLM) can surface from real-world discharge summaries.<n>LLM surfaced 3,460 candidate inconsistencies, affecting 69.7% of admissions.
  arXiv  Detail & Related papers  (2026-07-24T23:43:35Z)
• To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias [61.40435736418359]
  We introduce a unified framework that standardizes heterogeneous benchmarks to contrast isolated demographic assessments with forced-choice comparative settings.<n>We show that comparative settings act as aggressive catalysts for latent discrimination, a shift primarily driven by underspecified contexts.<n>Finally, we demonstrate that this comparative prejudice is a generalized phenomenon that scales positively with model size.
  arXiv  Detail & Related papers  (2026-06-23T13:53:50Z)
• Topological Ignorability for Structural Causal Effects Beyond Means [0.0]
  Interventions alter the structure of an outcome distribution rather than its mean.<n>Mean-based causal estimands such as the average treatment effect may miss important structural effects.<n>We introduce topological-geometrical causal metrics based on summaries of interventional outcome laws.
  arXiv  Detail & Related papers  (2026-05-31T11:56:53Z)
• ConfoundingSHAP: Quantifying confounding strength in causal inference [70.18600230387723]
  In causal inference, confounders are variables that influence both treatment decisions and outcomes.<n>We introduce ConfoundingSHAP, a Shapley-based method for attributing confounding strength to individual covariates.<n>We demonstrate the practical value across various datasets.
  arXiv  Detail & Related papers  (2026-05-11T13:19:33Z)
• Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis [57.78407973423517]
  We advocate uncertainty-aware subjectivity analysis, where models are expected to make predictions while expressing uncertainty that reflects human disagreement.<n>To operationalize this perspective, we propose a two-phase Disagreement Perception and Uncertainty Alignment framework.<n> Experiments on three subjectivity analysis tasks show that DPUA preserves task performance while better aligning model uncertainty with human disagreement.
  arXiv  Detail & Related papers  (2026-05-11T11:52:58Z)
• When Reviews Disagree: Fine-Grained Contradiction Analysis in Scientific Peer Reviews [25.43097618275061]
  RevCI is an expert-annotated benchmark of peer-review pairs with evidence-level contradiction annotations with graded intensity labels.<n> IMPACT is a structured multi-agent framework that integrates aspect-conditioned evidence extraction, deliberative reasoning, and adjudication.<n>TIDE is a small language model that predicts contradiction evidence and intensity in a single forward pass.
  arXiv  Detail & Related papers  (2026-05-11T08:20:58Z)
• Multiperspectivity as a Resource for Narrative Similarity Prediction [5.242380995471618]
  Different, equally valid readings of the same text can produce divergent interpretations and thus different similarity judgments.<n>We propose to incorporate this multiperspectivity in the decision making process of predictive systems.<n>Our experiments were conducted on the SemEval-2026 Task 4 dataset, where the system achieved an accuracy score of 0.705.
  arXiv  Detail & Related papers  (2026-03-23T15:32:40Z)
• Explanation Multiplicity in SHAP: Characterization and Assessment [28.413883186555438]
  Post-hoc explanations are widely used to justify, contest, and review automated decisions in high-stakes domains such as lending, employment, and healthcare.<n>In practice, however, SHAP explanations can differ substantially across repeated runs, even when the individual, prediction task, and trained model are held fixed.<n>We conceptualize and name this phenomenon explanation multiplicity: the existence of multiple, internally valid but substantively different explanations for the same decision.
  arXiv  Detail & Related papers  (2026-01-19T02:01:18Z)
• Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization [33.5861323022684]
  This paper introduces a flexible synthetic testbed that combines a statistical stream of generic tokens with an abstract factual stream of source-target token pairs.<n>We find that while higher contextual diversity delays in-distribution (ID) factual accuracy, its impact on out-of-distribution (OOD) factual generalization depends critically on contextual structure.
  arXiv  Detail & Related papers  (2025-10-17T17:58:01Z)
• Semantic Reformulation Entropy for Robust Hallucination Detection in QA Tasks [13.230578301939907]
  Existing entropy-based semantic-level uncertainty estimation methods are limited by sampling noise and unstable clustering of variable-length answers.<n>We propose Semantic Reformulation Entropy (SRE), which improves uncertainty estimation in two ways.
  arXiv  Detail & Related papers  (2025-09-22T07:38:45Z)
• Counterfactual Realizability [52.85109506684737]
  We introduce a formal definition of realizability, the ability to draw samples from a distribution, and then develop a complete algorithm to determine whether an arbitrary counterfactual distribution is realizable.<n>We illustrate the implications of this new framework for counterfactual data collection using motivating examples from causal fairness and causal reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-14T20:54:27Z)
• Estimating Commonsense Plausibility through Semantic Shifts [66.06254418551737]
  We propose ComPaSS, a novel discriminative framework that quantifies commonsense plausibility by measuring semantic shifts.<n> Evaluations on two types of fine-grained commonsense plausibility estimation tasks show that ComPaSS consistently outperforms baselines.
  arXiv  Detail & Related papers  (2025-02-19T06:31:06Z)
• Robust Representation Learning for Unreliable Partial Label Learning [86.909511808373]
  Partial Label Learning (PLL) is a type of weakly supervised learning where each training instance is assigned a set of candidate labels, but only one label is the ground-truth. This is known as Unreliable Partial Label Learning (UPLL) that introduces an additional complexity due to the inherent unreliability and ambiguity of partial labels. We propose the Unreliability-Robust Representation Learning framework (URRL) that leverages unreliability-robust contrastive learning to help the model fortify against unreliable partial labels effectively.
  arXiv  Detail & Related papers  (2023-08-31T13:37:28Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition [59.52434325897716]
  We propose a solution, named DMUE, to address the problem of annotation ambiguity from two perspectives. For the former, an auxiliary multi-branch learning framework is introduced to better mine and describe the latent distribution in the label space. For the latter, the pairwise relationship of semantic feature between instances are fully exploited to estimate the ambiguity extent in the instance space.
  arXiv  Detail & Related papers  (2021-04-01T03:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.