Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study

• URL: http://arxiv.org/abs/2607.20270v1
• Date: Wed, 22 Jul 2026 15:22:14 GMT
• Title: Which Values Do LLMs Confuse? A Schwartz-Based Recognition Study
• Abstract summary: We study controlled top-1 recognition over Schwartz's ten basic values.<n>We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol.<n>We find that models often locate the correct motivational region while ranking close alternatives unstably.
• Score: 32.505127447635864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.

Related papers

• Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models [49.45276299939287]
  Large Language Models (LLMs) increasingly handle complex subjective tasks.<n>LLMs fluctuate unpredictably under minor wording changes.<n> resolving this duality is critical for reliable AI alignment.
  arXiv  Detail & Related papers  (2026-09-15T03:35:04Z)
• Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence [0.7340017786387767]
  We show that prior scores, even when included only as context metadata, anchor judgments and systematically shift ratings toward their values.<n>On categorical industry data with human-labeled ground truth, anchored metadata blocks 48% of error corrections and flips 10.18% of correct judgments toward an assigned wrong label.
  arXiv  Detail & Related papers  (2026-08-26T14:41:24Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts [47.02978329409663]
  We propose a symmetric human-LLM evaluation framework, grounded in Q methodology, to measure value-structure alignment.<n>Under our protocol, humans and models sort an identical 140-item moral statement set into a shared nine-column forced distribution.<n>We evaluate 12 LLMs across four model families via 240 replicated Q-sorts at two temperature settings.
  arXiv  Detail & Related papers  (2026-06-20T08:15:41Z)
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias [0.0]
  Judge validation in practice relies on exact-match agreement, a metric that does not correct for chance and systematically overstates discriminative ability.<n>We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and approximately 541,000 individual judgments.
  arXiv  Detail & Related papers  (2026-06-17T19:37:13Z)
• Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges [2.8523456804049885]
  Safety judges are increasingly deployed to evaluate model outputs against evolving criteria.<n>Recent meta-evaluation work shows they remain brittle under prompt and rubric variation.<n>We propose a training strategy that combines instance-conditioned dynamic rubrics and a reliable-to-expressive curriculum.
  arXiv  Detail & Related papers  (2026-06-08T08:02:57Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Fact or Facsimile? Evaluating the Factual Robustness of Modern Retrievers [34.31192184496381]
  Dense retrievers and rerankers are central to retrieval-augmented generation (RAG) pipelines.<n>We evaluate how much factual competence these components inherit or lose from large language models (LLMs) they are based on.<n>For every embedding model, cosine-similarity scores between queries and correct completions are significantly higher than those for incorrect ones.
  arXiv  Detail & Related papers  (2025-08-28T04:13:51Z)
• Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values [76.70893269183684]
  Large Language Models (LLMs) achieve remarkable breakthroughs.<n> aligning their values with humans has become imperative for their responsible development.<n>There still lack evaluations of LLMs values that fulfill three desirable goals.
  arXiv  Detail & Related papers  (2025-01-13T05:53:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.