Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

• URL: http://arxiv.org/abs/2604.06210v2
• Date: Thu, 09 Apr 2026 02:01:43 GMT
• Title: Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
• Abstract summary: DOVE is a distributional evaluation framework that directly compares human-written text with LLM-generated outputs.<n>We show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.
• Score: 50.10594064510559
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As LLMs are globally deployed, aligning their cultural value orientations is critical for safety and user engagement. However, existing benchmarks face the Construct-Composition-Context ($C^3$) challenge: relying on discriminative, multiple-choice formats that probe value knowledge rather than true orientations, overlook subcultural heterogeneity, and mismatch with real-world open-ended generation. We introduce DOVE, a distributional evaluation framework that directly compares human-written text distributions with LLM-generated outputs. DOVE utilizes a rate-distortion variational optimization objective to construct a compact value-codebook from 10K documents, mapping text into a structured value space to filter semantic noise. Alignment is measured using unbalanced optimal transport, capturing intra-cultural distributional structures and sub-group diversity. Experiments across 12 LLMs show that DOVE achieves superior predictive validity, attaining a 31.56% correlation with downstream tasks, while maintaining high reliability with as few as 500 samples per culture.

Related papers

• Counterfactual Benchmarking and Training for Factuality Consistency and Order-Robust Grounded Reasoning in LLMs over Heterogeneous Knowledge [21.86609553205461]
  Large language models (LLMs) have increasingly supported response generation grounded in user-provided knowledge.<n>We introduce TKFQA, a factuality consistency benchmark comprising 10,130 question-answering (QA) pairs grounded in tables, texts, and knowledge graphs (KGs)<n>Each example is constructed from an explicit counterfactual reasoning chain, enabling the joint evaluation of answer correctness, reasoning-chain accuracy, and robustness to different input-order.
  arXiv  Detail & Related papers  (2026-08-08T00:46:32Z)
• Cultural Value Alignment Via Latent Activation Steering in Large Language Models [4.181458436156503]
  We propose a generalizable framework for cultural evaluation and intervention.<n>By extracting implicit token probabilities across 300 situational dilemmas, we bypass surface-level alignment.<n>We find substantial variation in adaptability and uncover a consistent phenomenon of latent entanglement.
  arXiv  Detail & Related papers  (2026-05-25T22:20:52Z)
• DVMap: Fine-Grained Pluralistic Value Alignment via High-Consensus Demographic-Value Mapping [49.402442592047855]
  Current Large Language Models (LLMs) typically rely on coarse-grained national labels for pluralistic value alignment.<n>We argue that resolving this limitation requires shifting from national labels to multi-dimensional demographic constraints.<n>We propose DVMap, a framework for fine-grained pluralistic value alignment.
  arXiv  Detail & Related papers  (2026-05-14T06:06:43Z)
• Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment [75.88231994139132]
  We show that even state-of-the-art models like GPT-4.1 achieve only 57.4% accuracy in predicting modal preferences.<n>We find significant pre-existing performance biases, where models better emulate young, male, Chinese, and Christian personas.
  arXiv  Detail & Related papers  (2026-04-14T15:06:13Z)
• Evidence-based Distributional Alignment for Large Language Models [58.65469623911573]
  We propose Evi-DA, an evidence-based alignment technique that improves the fidelity and robustness of LLM-based distribution estimation.<n>Given a target country, Evi-DA retrieves related World Values Survey items and their answer distributions, predicts a coarse Welzel value signature for each option, and infers the country-conditioned answer distribution in a structured format.
  arXiv  Detail & Related papers  (2026-03-03T03:34:06Z)
• Are Aligned Large Language Models Still Misaligned? [13.062124372682106]
  Mis-Align Bench is a unified benchmark for analyzing misalignment across safety, value, and cultural dimensions.<n> SAVACU is an English-aligned dataset of 382,424 misaligned samples spanning 112 domains (or labels)
  arXiv  Detail & Related papers  (2026-02-11T19:30:43Z)
• Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds [0.0]
  Large Language Models (LLMs) still produce gender-stereotyped language even in occupation-neutral contexts.<n>We compare six control techniques for bias mitigation: prompt-only, generate-and-filter, DFA-based Ctrl-G decoding, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Iterative Nullspace Projection (INLP)<n>SFT achieves 99.87 +- 0.15% compliance and high lexical diversity, while DPO, despite similar training stability, fails at 4.53 +- 0.82%.
  arXiv  Detail & Related papers  (2025-10-24T23:52:37Z)
• SCORE: A Semantic Evaluation Framework for Generative Document Parsing [2.5101597298392098]
  Multi-modal generative document parsing systems produce semantically correct yet structurally divergent outputs.<n>Conventional metrics-CER, WER, IoU, or TEDS-misclassify such diversity as error, penalizing valid interpretations and obscuring system behavior.<n>We introduce SCORE, an interpretation-agnostic framework that integrates (i) adjusted edit distance for robust content fidelity, (ii) token-level diagnostics to distinguish hallucinations from omissions, (iii) table evaluation with spatial tolerance and semantic alignment, and (iv) hierarchy-aware consistency checks.
  arXiv  Detail & Related papers  (2025-09-16T16:06:19Z)
• Leveraging Robust Optimization for LLM Alignment under Distribution Shifts [51.74394601039711]
  Preference alignment methods are increasingly critical for steering large language models to generate outputs consistent with human values.<n>We propose a novel distribution-aware optimization framework that improves preference alignment despite such shifts.
  arXiv  Detail & Related papers  (2025-04-08T09:14:38Z)
• Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Values [76.70893269183684]
  Large Language Models (LLMs) achieve remarkable breakthroughs.<n> aligning their values with humans has become imperative for their responsible development.<n>There still lack evaluations of LLMs values that fulfill three desirable goals.
  arXiv  Detail & Related papers  (2025-01-13T05:53:56Z)
• Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge [15.980606104936365]
  Large Language Models (LLMs) have revolutionized the landscape of machine learning, yet current benchmarks often fall short in capturing the diverse behavior of these models in real-world applications. Existing frameworks like Alpaca-Eval 2.0 LC citedubois2024lengthcontrolledalpacaevalsimpleway and Arena-Hard v0.1 citeli2024crowdsourced are limited by their focus on general-purpose queries and lack of diversity across domains such as law, medicine, and multilingual contexts. We introduce a novel data pipeline that curates, domain-specific evaluation sets tailored for LLM-as
  arXiv  Detail & Related papers  (2024-08-16T15:41:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.