Consensus Measures for Unstructured Biomedical Text Annotations

• URL: http://arxiv.org/abs/2608.03529v2
• Date: Wed, 05 Aug 2026 11:23:45 GMT
• Title: Consensus Measures for Unstructured Biomedical Text Annotations
• Abstract summary: We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks.<n>We show that soft reliability can be quantified using a variety of semantic equivalence measures.<n>We suggest measures based on natural language inference as a sensible compromise.
• Score: 5.516379363805726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Biomedical literature is increasingly mined for knowledge beyond the questions it was written to answer. Because the target concepts are not known in advance, annotators prefer open-ended labels, whose agreement is hard to quantify. We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks. Synthetic experiments show that soft reliability can be quantified using a variety of semantic equivalence measures, and that the choice of measure affects failure modes of the estimation. Embeddings are scalable, but limited when differentiating similar but distinct concepts. Large language models are promising, but limited by scalability for estimating agreement by chance. Finally, we suggest measures based on natural language inference as a sensible compromise.

Related papers

• ReportMedSAM: Guiding Segmentation Through Radiology Reports [44.07905920142891]
  ReportMedSAM is a report-driven framework that replaces discrete extraction with a learnable concept bank.<n>Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms.<n>ReportMedSAM effectively interprets free-form reports, achieves competitive segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.
  arXiv  Detail & Related papers  (2026-05-08T13:32:15Z)
• A Formal Framework for Uncertainty Analysis of Text Generation with Large Language Models [4.554894288663752]
  Generation of texts using Large Language Models (LLMs) is inherently uncertain.<n>We provide a formal framework for the measurement of uncertainty that takes these different aspects into account.
  arXiv  Detail & Related papers  (2026-03-27T12:44:37Z)
• QIME: Constructing Interpretable Medical Text Embeddings via Ontology-Grounded Questions [29.87717901839441]
  We propose QIME, an ontology-grounded framework for constructing interpretable medical text embeddings.<n>QIME generates semantically atomic questions that capture fine-grained distinctions in biomedical text.<n>We show that QIME consistently outperforms prior interpretable embedding methods and substantially narrows the gap to strong black-box biomedical encoders.
  arXiv  Detail & Related papers  (2026-03-02T10:18:06Z)
• The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure [98.71456610527598]
  Embedding-based similarity metrics can be influenced by spurious attributes like the text's source or language.<n>This paper shows that a debiasing algorithm that removes information about observed confounders from the encoder representations substantially reduces these biases at a minimal computational cost.
  arXiv  Detail & Related papers  (2025-07-01T23:17:12Z)
• Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness [106.52630978891054]
  We present a taxonomy of uncertainty specific to vision-language AI systems. We also introduce a new metric confidence-weighted accuracy, that is well correlated with both accuracy and calibration error.
  arXiv  Detail & Related papers  (2024-07-02T04:23:54Z)
• Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling [53.7117640028211]
  We present a geometry-constrained probabilistic modeling treatment to resolve the identified issues. We incorporate a suite of critical geometric properties to impose proper constraints on the layout of constructed embedding space. A spectral graph-theoretic method is devised to estimate the number of potential novel classes.
  arXiv  Detail & Related papers  (2024-03-02T00:56:05Z)
• CUE: An Uncertainty Interpretation Framework for Text Classifiers Built on Pre-Trained Language Models [28.750894873827068]
  We propose a novel framework, called CUE, which aims to interpret uncertainties inherent in the predictions of PLM-based models. By comparing the difference in predictive uncertainty between the perturbed and the original text representations, we are able to identify the latent dimensions responsible for uncertainty.
  arXiv  Detail & Related papers  (2023-06-06T11:37:46Z)
• Large-scale investigation of weakly-supervised deep learning for the fine-grained semantic indexing of biomedical literature [7.171698704686836]
  This study proposes a new method for the automated refinement of subject annotations at the level of MeSH concepts. The new method is evaluated on a large-scale retrospective scenario, based on concepts promoted to descriptors.
  arXiv  Detail & Related papers  (2023-01-23T10:33:22Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes.<n>We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach [3.2326259807823026]
  We analyze an alternative PMI-based metric to quantify biases in texts. It can be expressed as a function of conditional probabilities, which provides a simple interpretation in terms of word co-occurrences.
  arXiv  Detail & Related papers  (2021-04-13T19:34:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.