AnnotateThis: Analyzing a human-LLM system for annotating social media data with the concept of climate change mitigation pessimism

• URL: http://arxiv.org/abs/2606.10210v1
• Date: Mon, 08 Jun 2026 22:03:02 GMT
• Title: AnnotateThis: Analyzing a human-LLM system for annotating social media data with the concept of climate change mitigation pessimism
• Abstract summary: Large language models (LLMs) are increasingly being integrated into research.<n>LLMs have been shown to struggle with difficult and nuanced concepts such as those found in computational social science (CSS) research.<n>We develop AnnotateThis, a human-centered system for inspecting and improving LLM annotations.
• Score: 6.775474488255981
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly being integrated into research workflows. However, LLMs have been shown to struggle with difficult and nuanced concepts such as those found in computational social science (CSS) research. Within the CSS community, there has been a call for new systems to be developed which center humans in LLM-supported scientific workflows. We develop AnnotateThis, a human-centered system for inspecting and improving LLM annotations, a process we refer to as LLM grounding for a target concept. AnnotateThis is developed with both computational and social scientists to reflect existing workflows for data annotation. It includes a range of information features for users to interrogate the quality and reliability of LLM annotations. We evaluate our system in two settings. In the first, we assume a researcher may not have access to ground truth data and that users of AnnotateThis have limited prior knowledge of the concept they would like an LLM to annotate. That is, they may be conducting concept specification and LLM grounding simultaneously. In the second setting, we assume access to ground truth labels and that the concept is specified for a given annotation task; here, the task of LLM grounding is more straightforward. We find that in both settings users can improve the quality of LLM annotations with AnnotateThis and that their final annotations far surpass those created without human intervention. For example, when we evaluate with ground truth labels, we see an absolute improvement of 0.15 in F-Measure and 0.23 in accuracy over a fully automated state-of-the-art method for prompt refinement.

Related papers

• A Methodological Guide on Using Large Language Models for Text Annotation in the Social Sciences and Humanities with Python and R [1.1372969798040315]
  Large language models (LLMs) have become an essential tool for social science and humanities researchers.<n>This paper provides a comprehensive, step-by-step methodological guide for using LLMs for text annotation in SSH research.
  arXiv  Detail & Related papers  (2026-03-21T00:09:50Z)
• Can We Hide Machines in the Crowd? Quantifying Equivalence in LLM-in-the-loop Annotation Tasks [8.246529401043128]
  We aim to explore how labeling decisions -- by both humans and LLMs -- can be statistically evaluated across individuals.<n>We develop a statistical evaluation method based on Krippendorff's $alpha$, paired bootstrapping, and the Two One-Sided t-Tests (TOST) equivalence test procedure.<n>We apply this approach to two datasets -- MovieLens 100K and PolitiFact -- and find that the LLM is statistically indistinguishable from a human annotator in the former.
  arXiv  Detail & Related papers  (2025-10-08T05:17:33Z)
• Just Put a Human in the Loop? Investigating LLM-Assisted Annotation for Subjective Tasks [18.695435335031355]
  In subjective annotation tasks with multiple plausible answers, reviewing LLM outputs can change the label distribution.<n>We conducted a pre-registered experiment with 410 unique annotators and over 7,000 annotations.<n>We find that presenting crowdworkers with LLM-generated annotation suggestions did not make them faster, but did improve their self-reported confidence in the task.
  arXiv  Detail & Related papers  (2025-07-21T17:29:21Z)
• Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation [96.18720164390699]
  This paper explores the use of large language models (LLMs) for annotating document utility in training retrieval and retrieval-augmented generation (RAG) systems.<n>Our results show that LLM-generated annotations enhance out-of-domain retrieval performance and improve RAG outcomes compared to models trained solely on human annotations or downstream QA metrics.
  arXiv  Detail & Related papers  (2025-04-07T16:05:52Z)
• Latent Factor Models Meets Instructions: Goal-conditioned Latent Factor Discovery without Task Supervision [50.45597801390757]
  Instruct-LF is a goal-oriented latent factor discovery system.<n>It integrates instruction-following ability with statistical models to handle noisy datasets.
  arXiv  Detail & Related papers  (2025-02-21T02:03:08Z)
• LLM-Forest: Ensemble Learning of LLMs with Graph-Augmented Prompts for Data Imputation [50.375567142250446]
  Large language models (LLMs), trained on vast corpora, have shown strong potential in data generation.<n>We propose a novel framework, LLM-Forest, which introduces a "forest" of few-shot prompt learning LLM "trees" with their outputs aggregated via confidence-based weighted voting.<n>This framework is established on a new concept of bipartite information graphs to identify high-quality relevant neighboring entries with both feature and value granularity.
  arXiv  Detail & Related papers  (2024-10-28T20:42:46Z)
• Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring [21.7782670140939]
  Large language models (LLMs) have demonstrated strong potential in performing automatic scoring for constructed response assessments.<n>While constructed responses graded by humans are usually based on given grading rubrics, the methods by which LLMs assign scores remain largely unclear.<n>This paper uncovers the grading rubrics that LLMs used to score students' written responses to science tasks and their alignment with human scores.
  arXiv  Detail & Related papers  (2024-07-04T22:26:20Z)
• Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways [3.779027297957693]
  We test how prompt design impacts the compliance and accuracy of social science annotations. Our results show that LLM compliance and accuracy are highly prompt-dependent. This work serves as both a warning and practical guide for researchers and practitioners.
  arXiv  Detail & Related papers  (2024-06-17T18:01:43Z)
• PiCO: Peer Review in LLMs based on the Consistency Optimization [48.48819141999387]
  We use peer-review mechanisms to measure large language models (LLMs) automatically.<n>We formalize it as a constrained optimization problem, intending to maximize the consistency of each LLM's capabilities and scores.<n>We propose three metrics called PEN, CIN, and LIS to evaluate the gap in aligning human rankings.
  arXiv  Detail & Related papers  (2024-02-02T18:49:26Z)
• Can Large Language Models Transform Computational Social Science? [79.62471267510963]
  Large Language Models (LLMs) are capable of performing many language processing tasks zero-shot (without training data) This work provides a road map for using LLMs as Computational Social Science tools.
  arXiv  Detail & Related papers  (2023-04-12T17:33:28Z)
• Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models [68.37431984231338]
  Large language models (LLMs) have shown impressive results across a variety of tasks while requiring little or no direct supervision. We believe the ability of an LLM to an attribute to the text that it generates is likely to be crucial for both system developers and users in this setting.
  arXiv  Detail & Related papers  (2022-12-15T18:45:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.