Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer

• URL: http://arxiv.org/abs/2606.11018v1
• Date: Tue, 09 Jun 2026 15:55:55 GMT
• Title: Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer
• Abstract summary: Using non-English social media posts according to Schwartz's theory of basic human values, we investigate how different LLMs operationalize the expression of values in text.<n>We show that different LLMs produce different value interpretations. Iterative prompt calibration through error analysis reduces misattributions.<n>We derive targeted expert verification rules from recurrent error structures and use them during corpus annotation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Measuring subjective constructs in naturally occurring social media text requires annotation procedures that are theoretically grounded, empirically validated, and transferable to an encoder model for scalable prediction. Using non-English social media posts annotated according to Schwartz's theory of basic human values, we investigate how different LLMs, prompts, and instruction languages operationalize the expression of values in text. We argue that although texts may permit multiple plausible interpretations, theory-based value definitions can constrain interpretations and reduce spurious value attributions. Beyond precision, recall, and F1, we evaluate structural alignment between values, error structure, confidence-ambiguity relations, and annotation stability. We show that different LLMs produce different value interpretations. Iterative prompt calibration through error analysis reduces misattributions and improves alignment with expert annotations. We also derive targeted expert verification rules from recurrent error structures and use them during corpus annotation. Finally, we show that LLM annotations can be transferred to an encoder model through soft-label training, retaining theory-based value interpretations and information about uncertainty in value expression.

Related papers

• Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration [0.21410799064827235]
  Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data.<n>However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs.<n>This tension between scale and validity sits at the core of contemporary education data science.
  arXiv  Detail & Related papers  (2026-03-08T16:51:03Z)
• When Does Context Help? Error Dynamics of Contextual Information in Large Language Models [64.88201012057822]
  We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in large language models.<n>Our analysis characterizes contextual influence through output error dynamics.<n> Experiments across ICL, retrieval-augmented generation, and memory evolution validate our theory and motivate a principled context selection strategy.
  arXiv  Detail & Related papers  (2026-02-09T05:58:41Z)
• Enhancing LLM-Based Data Annotation with Error Decomposition [6.6544828402388445]
  Large language models offer a scalable alternative to human coding for data annotation tasks.<n>Their performance on subjective annotation tasks is less consistent and more prone to errors.<n>We propose a diagnostic evaluation paradigm that incorporates a human-in-the-loop step to separate task-inherent ambiguity from model-driven inaccuracies.
  arXiv  Detail & Related papers  (2026-01-17T05:43:17Z)
• Variance-Aware LLM Annotation for Strategy Research: Sources, Diagnostics, and a Protocol for Reliable Measurement [0.3228822469249803]
  Large language models (LLMs) offer strategy researchers powerful tools for annotating text at scale.<n>But treating LLM-generated labels as deterministic overlooks substantial instability.<n>We diagnose five variance sources: construct specification, interface effects, model preferences, output extraction, and system-level aggregation.
  arXiv  Detail & Related papers  (2025-12-02T18:02:20Z)
• Surrogate Representation Inference for Noisy Text and Image Annotations [0.0]
  Surrogate Representation Inference assumes that unstructured data fully mediates the relationship between human annotations and structured variables.<n>When multiple human annotations are available, SRI can further correct non-differential measurement errors that may exist in human annotations.
  arXiv  Detail & Related papers  (2025-09-15T20:09:21Z)
• Language Bottleneck Models: A Framework for Interpretable Knowledge Tracing and Beyond [55.984684518346924]
  We recast Knowledge Tracing as an inverse problem: learning the minimum natural-language summary that makes past answers explainable and future answers predictable.<n>Our Language Bottleneck Model (LBM) consists of an encoder LLM that writes an interpretable knowledge summary and a frozen decoder LLM that must reconstruct and predict student responses using only that summary text.<n> Experiments on synthetic arithmetic benchmarks and the large-scale Eedi dataset show that LBMs rival the accuracy of state-of-the-art KT and direct LLM methods while requiring orders-of-magnitude fewer student trajectories.
  arXiv  Detail & Related papers  (2025-06-20T13:21:14Z)
• MoRE-LLM: Mixture of Rule Experts Guided by a Large Language Model [54.14155564592936]
  We propose a Mixture of Rule Experts guided by a Large Language Model (MoRE-LLM)<n>MoRE-LLM steers the discovery of local rule-based surrogates during training and their utilization for the classification task.<n>LLM is responsible for enhancing the domain knowledge alignment of the rules by correcting and contextualizing them.
  arXiv  Detail & Related papers  (2025-03-26T11:09:21Z)
• Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment [53.17596274334017]
  We evaluate the correctness of the detailed inference patterns of an LLM behind its seemingly correct outputs.<n>Experiments show that even when the language generation results appear correct, a significant portion of the inference patterns used by the LLM for the legal judgment may represent misleading or irrelevant logic.
  arXiv  Detail & Related papers  (2024-10-06T08:33:39Z)
• Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations [48.07182711678573]
  ASTrust generates explanations grounded in the relationship between model confidence and syntactic structures of programming languages. We develop an automated visualization that illustrates the aggregated model confidence scores superimposed on sequence, heat-map, and graph-based visuals of syntactic structures from ASTs.
  arXiv  Detail & Related papers  (2024-07-12T04:38:28Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.