Rater Cohesion and Quality from a Vicarious Perspective

• URL: http://arxiv.org/abs/2408.08411v2
• Date: Fri, 4 Oct 2024 19:51:00 GMT
• Title: Rater Cohesion and Quality from a Vicarious Perspective
• Authors: Deepak Pandita, Tharindu Cyril Weerasooriya, Sujan Dutta, Sarah K. Luger, Tharindu Ranasinghe, Ashiqur R. KhudaBukhsh, Marcos Zampieri, Christopher M. Homan,
• Abstract summary: Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.
• Score: 22.445283423317754
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human feedback is essential for building human-centered AI systems across domains where disagreement is prevalent, such as AI safety, content moderation, or sentiment analysis. Many disagreements, particularly in politically charged settings, arise because raters have opposing values or beliefs. Vicarious annotation is a method for breaking down disagreement by asking raters how they think others would annotate the data. In this paper, we explore the use of vicarious annotation with analytical methods for moderating rater disagreement. We employ rater cohesion metrics to study the potential influence of political affiliations and demographic backgrounds on raters' perceptions of offense. Additionally, we utilize CrowdTruth's rater quality metrics, which consider the demographics of the raters, to score the raters and their annotations. We study how the rater quality metrics influence the in-group and cross-group rater cohesion across the personal and vicarious levels.

Related papers

• Diverging Preferences: When do Annotators Disagree and do Models Know? [92.24651142187989]
  We develop a taxonomy of disagreement sources spanning 10 categories across four high-level classes. We find that the majority of disagreements are in opposition with standard reward modeling approaches. We develop methods for identifying diverging preferences to mitigate their influence on evaluation and training.
  arXiv  Detail & Related papers  (2024-10-18T17:32:22Z)
• Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback [8.04095222893591]
  We find significant gaps in fairness preferences depending on the race, age, political stance, educational level, and LGBTQ+ identity of annotators. We also demonstrate that demographics mentioned in text have a strong influence on how users perceive individual fairness in moderation.
  arXiv  Detail & Related papers  (2024-06-09T19:42:25Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Investigating the Nature of Disagreements on Mid-Scale Ratings: A Case Study on the Abstractness-Concreteness Continuum [8.086165096687772]
  Humans tend to strongly agree on ratings on a scale for extreme cases, but judgements on mid-scale words exhibit more disagreement. Our study focuses on concreteness ratings and implements correlations and supervised classification to identify salient multi-modal characteristics of mid-scale words. Our results suggest to either fine-tune or filter mid-scale target words before utilising them.
  arXiv  Detail & Related papers  (2023-11-08T09:52:58Z)
• OpinSummEval: Revisiting Automated Evaluation for Opinion Summarization [52.720711541731205]
  We present OpinSummEval, a dataset comprising human judgments and outputs from 14 opinion summarization models. Our findings indicate that metrics based on neural networks generally outperform non-neural ones.
  arXiv  Detail & Related papers  (2023-10-27T13:09:54Z)
• Do Differences in Values Influence Disagreements in Online Discussions? [4.128725138940779]
  We show how state-of-the-art models can be used for estimating values in online discussions. We evaluate the estimated value profiles based on human-annotated agreement labels. We find that the dissimilarity of value profiles correlates with disagreement in specific cases.
  arXiv  Detail & Related papers  (2023-10-24T12:00:59Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs) We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric. Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Fairness meets Cross-Domain Learning: a new perspective on Models and Metrics [80.07271410743806]
  We study the relationship between cross-domain learning (CD) and model fairness. We introduce a benchmark on face and medical images spanning several demographic groups as well as classification and localization tasks. Our study covers 14 CD approaches alongside three state-of-the-art fairness algorithms and shows how the former can outperform the latter.
  arXiv  Detail & Related papers  (2023-03-25T09:34:05Z)
• Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information [11.227630261409706]
  We study whether the text of a task and annotators' demographic background information can be used to estimate the level of disagreement among annotators. Our results show that knowing annotators' demographic information, like gender, ethnicity, and education level, helps predict disagreements.
  arXiv  Detail & Related papers  (2023-01-12T14:04:53Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.