Data Annotations as Pedagogical Hints: From Subjective Labels to Critical Thinking

• URL: http://arxiv.org/abs/2607.20149v1
• Date: Wed, 22 Jul 2026 13:51:56 GMT
• Title: Data Annotations as Pedagogical Hints: From Subjective Labels to Critical Thinking
• Abstract summary: Machine learning courses often use pre-labeled datasets, hiding the subjectivity of human annotation.<n>This creates students with an overly trusting view of AI data and models, undervaluing interpretive diversity.<n>We investigated whether manual data annotation tasks teach students about subjective labeling.
• Score: 15.463313028798767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning courses often use pre-labeled datasets, hiding the subjectivity of human annotation. This creates students with an overly trusting view of AI data and models, undervaluing interpretive diversity. We investigated whether manual data annotation tasks teach students about subjective labeling. Study Design: An annotation activity was implemented at two universities: Fontys (Netherlands) and IT University Copenhagen (Denmark). Students annotated skin lesion images for hair coverage on a 3-point scale. Surveys were collected from 43 participants measuring their understanding of annotation ambiguity, data quality, bias, fairness, implementation barriers, and pedagogical effectiveness. Key Findings: Self-reported familiarity with course content increased substantially across all concepts. Most students recognised that personal interpretation affects annotations. Students rated the activity as more effective than traditional lectures for understanding bias. Participants were motivated to learn more. Main Drawbacks: Emotional unease from viewing medical images was the primary issue. Many students still requested clearer guidelines to reduce disagreement, suggesting they hadn't internalised that disagreement from different perspectives is a learning feature, not a bug. Recommendations for Future Iterations: Ensure sufficient interpretive ambiguity in materials. Reduce repetitive annotation workload. Mitigate emotional unease from sensitive content. Explicitly frame disagreement as a learning opportunity rather than a problem to solve. Manual data annotations effectively teach students that human judgment shapes model behavior and that disagreement reflects domain complexity, not just noise.

Related papers

• Learning Moral Diversity: Modelling Individual Perspectives in Moral Classification of Texts [7.927346467214517]
  We extend a pretrained language model with a layer that learns annotator-specific features.<n>Our model improves predictions of individual annotations and yields representations that reveal meaningful insights into annotators' moral perspectives.
  arXiv  Detail & Related papers  (2026-06-22T02:19:23Z)
• Students' Perceptions of Peer Grading [0.8941624592392746]
  We conducted a mixed-source thematic analysis of literature and student discussions on Reddit.<n>The study synthesized evidence from 107 papers, 114 Reddit posts, and 300 comments.<n>Students view peer grading as both beneficial and problematic.
  arXiv  Detail & Related papers  (2026-06-13T02:46:09Z)
• Disentangling Learning from Judgment: Representation Learning for Open Response Analytics [0.0]
  Open-ended responses are central to learning, yet automated scoring often conflates what students wrote with how teachers grade.<n>We present an analytics-first framework that separates content signals from rater tendencies, making judgments visible and auditable via analytics.
  arXiv  Detail & Related papers  (2025-12-30T02:06:28Z)
• Can Reasoning Help Large Language Models Capture Human Annotator Disagreement? [84.32752330104775]
  Variation in human annotation (i.e., disagreements) is common in NLP.<n>We evaluate the influence of different reasoning settings on Large Language Model disagreement modeling.<n>Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling.
  arXiv  Detail & Related papers  (2025-06-24T09:49:26Z)
• FedIA: Federated Medical Image Segmentation with Heterogeneous Annotation Completeness [30.780654470392125]
  Federated learning has emerged as a compelling paradigm for medical image segmentation. This paper highlights a prevalent challenge in medical practice: incomplete annotations. We introduce a novel solution, named FedIA, to tackle this issue.
  arXiv  Detail & Related papers  (2024-07-02T14:08:55Z)
• Understanding Before Recommendation: Semantic Aspect-Aware Review Exploitation via Large Language Models [53.337728969143086]
  Recommendation systems harness user-item interactions like clicks and reviews to learn their representations. Previous studies improve recommendation accuracy and interpretability by modeling user preferences across various aspects and intents. We introduce a chain-based prompting approach to uncover semantic aspect-aware interactions.
  arXiv  Detail & Related papers  (2023-12-26T15:44:09Z)
• Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Tasks [9.110872603799839]
  Supervised classification heavily depends on datasets annotated by humans. In subjective tasks such as toxicity classification, these annotations often exhibit low agreement among raters. In this work, we propose textbfAnnotator Awares for Texts (AART) for subjective classification tasks.
  arXiv  Detail & Related papers  (2023-11-16T10:18:32Z)
• How are Prompts Different in Terms of Sensitivity? [50.67313477651395]
  We present a comprehensive prompt analysis based on the sensitivity of a function. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output. We introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding.
  arXiv  Detail & Related papers  (2023-11-13T10:52:01Z)
• Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem [60.0878532426877]
  We propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents. The experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2022-07-24T23:50:52Z)
• Annotation Curricula to Implicitly Train Non-Expert Annotators [56.67768938052715]
  voluntary studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations. We propose annotation curricula, a novel approach to implicitly train annotators.
  arXiv  Detail & Related papers  (2021-06-04T09:48:28Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
  We propose a novel disentanglement approach to invariant representation problem. We enforce the meaningful representation to be agnostic to sensitive information by entropy. The proposed approach is evaluated on five publicly available datasets.
  arXiv  Detail & Related papers  (2020-03-12T11:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.