Subjective Logic Encodings

• URL: http://arxiv.org/abs/2502.12225v2
• Date: Thu, 20 Mar 2025 15:52:29 GMT
• Title: Subjective Logic Encodings
• Authors: Jake Vasilakes, Chrysoula Zerva, Sophia Ananiadou,
• Abstract summary: Data perspectivism seeks to leverage inter-annotator disagreement to learn models.<n>Subjective Logic SLEs is a framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators.
• Score: 20.458601113219697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many existing approaches for learning from labeled data assume the existence of gold-standard labels. According to these approaches, inter-annotator disagreement is seen as noise to be removed, either through refinement of annotation guidelines, label adjudication, or label filtering. However, annotator disagreement can rarely be totally eradicated, especially on more subjective tasks such as sentiment analysis or hate speech detection where disagreement is natural. Therefore, a new approach to learning from labeled data, called data perspectivism, seeks to leverage inter-annotator disagreement to learn models that stay true to the inherent uncertainty of the task by treating annotations as opinions of the annotators, rather than gold-standard facts. Despite this conceptual grounding, existing methods under data perspectivism are limited to using disagreement as the sole source of annotation uncertainty. To expand the possibilities of data perspectivism, we introduce Subjective Logic Encodings (SLEs), a flexible framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators. Based on Subjective Logic Theory, SLEs encode labels as Dirichlet distributions and provide principled methods for encoding and aggregating various types of annotation uncertainty -- annotator confidence, reliability, and disagreement -- into the targets. We show that SLEs are a generalization of other types of label encodings as well as how to estimate models to predict SLEs using a distribution matching objective.

Related papers

• From Disagreement to Understanding: The Case for Ambiguity Detection in NLI [0.0]
  We show that content-based ambiguity offers a process-independent signal of divergent human perspectives.<n>A key limitation is the lack of annotated datasets for ambiguity and subtypes.<n>We propose addressing this gap through new annotated resources and unsupervised approaches to ambiguity detection.
  arXiv  Detail & Related papers  (2025-07-20T20:27:35Z)
• When LLMs Disagree: Diagnosing Relevance Filtering Bias and Retrieval Divergence in SDG Search [0.0]
  Large language models (LLMs) are increasingly used to assign document relevance labels in information retrieval pipelines.<n>LLMs often disagree on borderline cases, raising concerns about how such disagreement affects downstream retrieval.<n>We show that model disagreement is systematic, not random.<n>We propose using classification disagreement as an object of analysis in retrieval evaluation, particularly in policy-relevant or thematic search tasks.
  arXiv  Detail & Related papers  (2025-07-02T20:53:51Z)
• Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Prompts [26.415262737856967]
  We explore label verification in contexts using Large Language Models (LLMs)<n>We propose the Label-in-a-Haystack Rectification (LiaHR) framework for subjective label correction.<n>This approach can be integrated into annotation pipelines to enhance signal-to-noise ratios.
  arXiv  Detail & Related papers  (2025-05-22T18:55:22Z)
• Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks [9.110872603799839]
  Supervised classification heavily depends on datasets annotated by humans. In subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. In this work, we propose textbfAnnotator Awares for Texts (AART) for subjective classification tasks.
  arXiv  Detail & Related papers  (2023-11-16T10:18:32Z)
• Unsupervised Pretraining for Fact Verification by Language Model Distillation [4.504050940874427]
  We propose SFAVEL (Self-supervised Fact Verification via Language Model Distillation), a novel unsupervised pretraining framework. It distils self-supervised features into high-quality claim-fact alignments without the need for annotations. This is enabled by a novel contrastive loss function that encourages features to attain high-quality claim and evidence alignments.
  arXiv  Detail & Related papers  (2023-09-28T15:53:44Z)
• Robust Representation Learning for Unreliable Partial Label Learning [86.909511808373]
  Partial Label Learning (PLL) is a type of weakly supervised learning where each training instance is assigned a set of candidate labels, but only one label is the ground-truth. This is known as Unreliable Partial Label Learning (UPLL) that introduces an additional complexity due to the inherent unreliability and ambiguity of partial labels. We propose the Unreliability-Robust Representation Learning framework (URRL) that leverages unreliability-robust contrastive learning to help the model fortify against unreliable partial labels effectively.
  arXiv  Detail & Related papers  (2023-08-31T13:37:28Z)
• Concept-Based Explanations to Test for False Causal Relationships Learned by Abusive Language Classifiers [7.022948483613113]
  We consider three well-known abusive language classifiers trained on large English datasets. We first examine the unwanted dependencies learned by the classifiers by assessing their accuracy on a challenge set across all decision thresholds. We then introduce concept-based explanation metrics to assess the influence of the concept on the labels.
  arXiv  Detail & Related papers  (2023-07-04T19:57:54Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs) We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Adaptive Negative Evidential Deep Learning for Open-set Semi-supervised Learning [69.81438976273866]
  Open-set semi-supervised learning (Open-set SSL) considers a more practical scenario, where unlabeled data and test data contain new categories (outliers) not observed in labeled data (inliers) We introduce evidential deep learning (EDL) as an outlier detector to quantify different types of uncertainty, and design different uncertainty metrics for self-training and inference. We propose a novel adaptive negative optimization strategy, making EDL more tailored to the unlabeled dataset containing both inliers and outliers.
  arXiv  Detail & Related papers  (2023-03-21T09:07:15Z)
• Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective [89.5370481649529]
  We propose a label distribution perspective for PU learning in this paper. Motivated by this, we propose to pursue the label distribution consistency between predicted and ground-truth label distributions. Experiments on three benchmark datasets validate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-12-06T07:38:29Z)
• Meta Objective Guided Disambiguation for Partial Label Learning [44.05801303440139]
  We propose a novel framework for partial label learning with meta objective guided disambiguation (MoGD) MoGD aims to recover the ground-truth label from candidate labels set by solving a meta objective on a small validation set. The proposed method can be easily implemented by using various deep networks with the ordinary SGD.
  arXiv  Detail & Related papers  (2022-08-26T06:48:01Z)
• Resolving label uncertainty with implicit posterior models [71.62113762278963]
  We propose a method for jointly inferring labels across a collection of data samples. By implicitly assuming the existence of a generative model for which a differentiable predictor is the posterior, we derive a training objective that allows learning under weak beliefs.
  arXiv  Detail & Related papers  (2022-02-28T18:09:44Z)
• Debiased Contrastive Learning [64.98602526764599]
  We develop a debiased contrastive objective that corrects for the sampling of same-label datapoints. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2020-07-01T04:25:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.