Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs

• URL: http://arxiv.org/abs/2411.05040v1
• Date: Wed, 06 Nov 2024 18:51:04 GMT
• Title: Bottom-Up and Top-Down Analysis of Values, Agendas, and Observations in Corpora and LLMs
• Authors: Scott E. Friedman, Noam Benkler, Drisana Mosaphir, Jeffrey Rye, Sonja M. Schmer-Galunder, Micah Goldwater, Matthew McLure, Ruta Wheelock, Jeremy Gottlieb, Robert P. Goldman, Christopher Miller,
• Abstract summary: Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives. We seek to characterize socio-cultural values that they express, for reasons of safety, accuracy, inclusion, and cultural fidelity.
• Score: 1.3119775978504942
• License:
• Abstract: Large language models (LLMs) generate diverse, situated, persuasive texts from a plurality of potential perspectives, influenced heavily by their prompts and training data. As part of LLM adoption, we seek to characterize - and ideally, manage - the socio-cultural values that they express, for reasons of safety, accuracy, inclusion, and cultural fidelity. We present a validated approach to automatically (1) extracting heterogeneous latent value propositions from texts, (2) assessing resonance and conflict of values with texts, and (3) combining these operations to characterize the pluralistic value alignment of human-sourced and LLM-sourced textual data.

Related papers

• LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Output [8.435090588116973]
  We propose the LLM-GLOBE benchmark for evaluating the cultural value systems of LLMs. We then leverage the benchmark to compare the values of Chinese and US LLMs. Our methodology includes a novel "LLMs-as-a-Jury" pipeline which automates the evaluation of open-ended content.
  arXiv  Detail & Related papers  (2024-11-09T01:38:55Z)
• Measuring Human and AI Values based on Generative Psychometrics with Large Language Models [13.795641564238434]
  In recent advances in AI, large language models (LLMs) have emerged as both tools and subjects of value measurement. This work introduces Generative Psychometrics for Values (GPV), a data-driven value measurement paradigm grounded in text-revealed selective perceptions.
  arXiv  Detail & Related papers  (2024-09-18T16:26:22Z)
• CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses [34.77031649891843]
  We introduce CLAVE, a novel framework which integrates two complementary Large Language Models (LLMs) This dual-model approach enables calibration with any value systems using 100 human-labeled samples per value type. We present ValEval, a comprehensive dataset comprising 13k+ (text,value,label) 12+s across diverse domains, covering three major value systems.
  arXiv  Detail & Related papers  (2024-07-15T13:51:37Z)
• CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models [59.22460740026037]
  "CIVICS: Culturally-Informed & Values-Inclusive Corpus for Societal impacts" dataset is designed to evaluate the social and cultural variation of Large Language Models (LLMs) We create a hand-crafted, multilingual dataset of value-laden prompts which address specific socially sensitive topics, including LGBTQI rights, social welfare, immigration, disability rights, and surrogacy.
  arXiv  Detail & Related papers  (2024-05-22T20:19:10Z)
• Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches [69.73783026870998]
  This work proposes a novel framework, ValueLex, to reconstruct Large Language Models' unique value system from scratch. Based on Lexical Hypothesis, ValueLex introduces a generative approach to elicit diverse values from 30+ LLMs. We identify three core value dimensions, Competence, Character, and Integrity, each with specific subdimensions, revealing that LLMs possess a structured, albeit non-human, value system.
  arXiv  Detail & Related papers  (2024-04-19T09:44:51Z)
• CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge [69.82940934994333]
  We introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build challenging evaluation dataset. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions. CULTURALBENCH-V0.1 is a compact yet high-quality evaluation dataset with users' red-teaming attempts.
  arXiv  Detail & Related papers  (2024-04-10T00:25:09Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties [68.66719970507273]
  Value pluralism is the view that multiple correct values may be held in tension with one another. As statistical learners, AI systems fit to averages by default, washing out potentially irreducible value conflicts. We introduce ValuePrism, a large-scale dataset of 218k values, rights, and duties connected to 31k human-written situations.
  arXiv  Detail & Related papers  (2023-09-02T01:24:59Z)
• Heterogeneous Value Alignment Evaluation for Large Language Models [91.96728871418]
  Large Language Models (LLMs) have made it crucial to align their values with those of humans. We propose a Heterogeneous Value Alignment Evaluation (HVAE) system to assess the success of aligning LLMs with heterogeneous values.
  arXiv  Detail & Related papers  (2023-05-26T02:34:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.