Human Label Variation in Implicit Discourse Relation Recognition

• URL: http://arxiv.org/abs/2602.22723v1
• Date: Thu, 26 Feb 2026 07:56:24 GMT
• Title: Human Label Variation in Implicit Discourse Relation Recognition
• Abstract summary: Implicit Discourse Relation Recognition (IDRR) is a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias.<n>Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced.<n>Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.
• Score: 14.762949592713914
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: There is growing recognition that many NLP tasks lack a single ground truth, as human judgments reflect diverse perspectives. To capture this variation, models have been developed to predict full annotation distributions rather than majority labels, while perspectivist models aim to reproduce the interpretations of individual annotators. In this work, we compare these approaches on Implicit Discourse Relation Recognition (IDRR), a highly ambiguous task where disagreement often arises from cognitive complexity rather than ideological bias. Our experiments show that existing annotator-specific models perform poorly in IDRR unless ambiguity is reduced, whereas models trained on label distributions yield more stable predictions. Further analysis indicates that frequent cognitively demanding cases drive inconsistency in human interpretation, posing challenges for perspectivist modeling in IDRR.

Related papers

• How Unlikely Is "Unlikely"? Assessing Verbal Probability Perception Across Large Language Models [1.435579512127234]
  We present a systematic cross-model evaluation using a word-to-number mapping task grounded in established human benchmarks.<n>Eleven uncertainty expressions were presented to 19 models under two conditions, forced single-number response and explanation elicitation.<n>Models show a systematic upward bias for negative expressions such as unlikely'' and improbable''
  arXiv  Detail & Related papers  (2026-08-26T19:00:41Z)
• Aligning LLM Uncertainty with Human Disagreement in Subjectivity Analysis [57.78407973423517]
  We advocate uncertainty-aware subjectivity analysis, where models are expected to make predictions while expressing uncertainty that reflects human disagreement.<n>To operationalize this perspective, we propose a two-phase Disagreement Perception and Uncertainty Alignment framework.<n> Experiments on three subjectivity analysis tasks show that DPUA preserves task performance while better aligning model uncertainty with human disagreement.
  arXiv  Detail & Related papers  (2026-05-11T11:52:58Z)
• Scaling Reasoning Hop Exposes Weaknesses: Demystifying and Improving Hop Generalization in Large Language Models [66.36240676392502]
  Chain-of-thought (CoT) reasoning has become the standard paradigm for enabling Large Language Models (LLMs) to solve complex problems.<n>Recent studies reveal a sharp performance drop in reasoning hop generalization scenarios.<n>We propose test-time correction of reasoning, a lightweight intervention method that dynamically identifies and deactivates ep heads in the reasoning process.
  arXiv  Detail & Related papers  (2026-01-29T03:24:32Z)
• Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems [3.011820285006942]
  This study proposes a new multi-perspective approach using soft labels to encourage the development of perspective aware models.<n>We conduct an analysis across diverse subjective text classification tasks, including hate speech, irony, abusive language, and stance detection.<n>Results show that the multi-perspective approach better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD)<n>Our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts.
  arXiv  Detail & Related papers  (2025-06-25T07:53:36Z)
• Improving Group Robustness on Spurious Correlation via Evidential Alignment [26.544938760265136]
  Deep neural networks often learn and rely on spurious correlations, i.e., superficial associations between non-causal features and the targets.<n>Existing methods typically mitigate this issue by using external group annotations or auxiliary deterministic models.<n>We propose Evidential Alignment, a novel framework that leverages uncertainty quantification to understand the behavior of the biased models.
  arXiv  Detail & Related papers  (2025-06-12T22:47:21Z)
• Preference Learning for AI Alignment: a Causal Perspective [55.2480439325792]
  We frame this problem in a causal paradigm, providing the rich toolbox of causality to identify persistent challenges.<n>Inheriting from the literature of causal inference, we identify key assumptions necessary for reliable generalisation.<n>We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness.
  arXiv  Detail & Related papers  (2025-06-06T10:45:42Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Contrastive Representation Modeling for Anomaly Detection [0.21427777919040417]
  We propose a structured contrastive objective that redefines positive and negative relationships during training, promoting these properties without requiring explicit anomaly labels.<n>We extend this framework with a patch-based learning and evaluation strategy specifically designed to improve the detection of localized anomalies in industrial settings.
  arXiv  Detail & Related papers  (2025-01-09T10:33:16Z)
• Multi-Perspective Stance Detection [1.3897429860422184]
  Multi-perspective approach yields better classification performance than the baseline which uses the single label.<n>This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.
  arXiv  Detail & Related papers  (2024-11-13T16:30:41Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [104.60508550106618]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs)<n>We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features.<n>We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• Identifiable Latent Polynomial Causal Models Through the Lens of Change [82.14087963690561]
  Causal representation learning aims to unveil latent high-level causal representations from observed low-level data.<n>One of its primary tasks is to provide reliable assurance of identifying these latent causal models, known as identifiability.
  arXiv  Detail & Related papers  (2023-10-24T07:46:10Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Uncertainty-Aware Pedestrian Trajectory Prediction via Distributional Diffusion [26.715578412088327]
  We present a model-agnostic uncertainty-aware pedestrian trajectory prediction framework. Unlike previous studies, we translate the predictiveity to explicit distributions, allowing it to generate plausible future trajectories. Our framework is compatible with different neural net architectures.
  arXiv  Detail & Related papers  (2023-03-15T04:58:43Z)
• Aleatoric uncertainty for Errors-in-Variables models in deep regression [0.48733623015338234]
  We show how the concept of Errors-in-Variables can be used in Bayesian deep regression. We discuss the approach along various simulated and real examples.
  arXiv  Detail & Related papers  (2021-05-19T12:37:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.